Docker driver client log grows indefinitely

[floriandotorg]

Describe the bug
The loki docker driver creates a giant log file at /var/lib/docker/plugins/7c1139a6678d7f0f3c62a3525c9515bb583085bd4d0723d3cc0a0a1a653dda01/rootfs/var/log/docker/a2507b4ccf55b42da52786e62cbb9c7268d3f4454437c5f127cc16a083e50f31. Not matter the config, the file grows indefinitely.

Settings

{
  "log-driver": "loki",
  "log-opts": {
    "loki-url": "http://localhost:3100/loki/api/v1/push",
    "loki-retries": "2",
    "loki-max-backoff": "800ms",
    "loki-timeout": "3s",
    "keep-file": "false",
    "max-size": "1g",
    "max-file": "1",
    "mode": "non-blocking",
    "max-buffer-size": "4m"
  }
}

To Reproduce
Steps to reproduce the behavior:

1. Setup docker driver, according to this documentation: https://grafana.com/docs/loki/latest/send-data/docker-driver/
2. Use daemon.json from above.
3. Deploy a docker container that creates a lot of log output.

Expected behavior
The log file is kept under 1gb as configured.

Environment:

• Infrastructure: bare-metal
• Deployment tool: docker

Screenshots, Promtail config, or terminal output

Thanks everyone, for your amazing work <3

[JoaquinJimenezGarcia]

Hey there!

Sorry if I left something behind but I am new contributing to open source projects :( I tried to replicate your problem and had no luck. Just a few questions: are you sending the logs directly to Loki or using Promtail?

I am ingesting the logs from Promtail, and when I have a log file over 1GiB Promtail return an error and Loki doesn't ingest the log.

loki_1 | level=error ts=2025-02-23T14:53:55.028947698Z caller=manager.go:50 component=distributor path=write msg="write operation failed" details="Ingestion rate limit exceeded for user fake (limit: 4194304 bytes/sec) while attempting to ingest '4094' lines totaling '1048515' bytes, reduce log volume or contact your Loki administrator to see if the limit can be increased" org_id=fake

What you showed in your description doesn't look like the logs Loki ingests. Also, the limitation is on Loki to save the logs, not on the log file itself. For me it looks like you should take care of your file not growing bigger than 1GiB.

Once again, sorry if I am wrong but that's what I could see on my end replicating your steps.

[floriandotorg]

Hey @JoaquinJimenezGarcia, first thanks for taking the time to look into this!

No, I'm not using promtail, I'm using the loki docker driver to send the logs directly, so there are no files to grow.

Loki is running as a docker container as well, could that cause the problem?