Skip to content

if loki is not reachable and loki-docker-driver is activated, containers apps stops and cannot be stopped/killed #2361

@badsmoke

Description

@badsmoke

Describe the bug
we have installed the loki-docker-driver on all our devices.
The loki server on an extra server, if the loki-server is updated/restarted or just not reachable then after a short time all containers get stuck (docker logs does not update anymore).
If the loki-server is not reachable, the containers can neither be stopped/kill nor restarted.

To Reproduce
Steps to reproduce the behavior:

  1. start loki server (server)
  2. install loki-docker-driver on another system (can also be tested on one and the same system) (client)
    2.1. /etc/docker/daemon.json { "live-restore": true, "log-driver": "loki", "log-opts": { "loki-url": "http://loki:3100/api/prom/push", "mode": "non-blocking", "loki-batch-size": "400", "max-size": "1g" } }
  3. docker run --rm --name der-container -d debian /bin/sh -c "while true; do date >> /tmp/ts ; seq 0 1000000; sleep 1 ; done"(client)
  4. docker exec -it der-container tail -f /tmp/ts
    shows every second the time (client)
  5. docker logs -f der-container show numbers from 0-1000000 (client)
  6. stop loki server (server)
  7. you will see that the outputs on the system stop with the loci-driver and that you cannot stop the container (client)
  8. docker stop der-container (client)

Expected behavior
A clear and concise description of what you expected to happen.
I would like all containers to continue to run as desired even if the loci is not accessible.
That man container can start/stop even if loki is not reachable

Environment:

  • Infrastructure: [bare-metal, laptop, VMs]
  • Deployment tool: [docker-compose]

Screenshots, Promtail config, or terminal output
loki-docker-driver version: loki-docker-driver:master-616771a (from then on the driver option "non-blocking" is supported)
loki server: 1.5.0

I am very grateful for any help, this problem has caused our whole system to collapse

Metadata

Metadata

Labels

component/docker-driverkeepaliveAn issue or PR that will be kept alive and never marked as stale.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions