Tecnativa / Tecnativa/docker-socket-proxy

Container never stops while an /events stream is open: soft-stop waits forever, no hard-stop-after

Open
#181 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.8k
Forks
207
PR merge metrics
No merged PRs in 30d

Description

Summary

With a client holding /events open — Traefik's Docker provider does this permanently — the container never stops on its own. Not slowly: never. docker stop always ends in SIGKILL and exit 137 at whatever deadline it is given.

This is a consequence of the fix for #62, not a regression in it. The dedicated docker-events backend with timeout server 0 is right — an event stream should not be cut every ten minutes — but on SIGTERM HAProxy performs a soft stop and waits for established sessions to end, and that session has no end. Nothing bounds the wait, because hard-stop-after is not set.

HAProxy warns about this itself, at every boot:

[WARNING] (12) : missing timeouts for backend 'docker-events'.
  | While not properly invalid, you will certainly encounter various problems
  | with such a configuration. To fix this, please ensure that all following
  | timeouts are set to a non-zero value: 'client', 'connect', 'server'.

Reproduction

v0.4.2, HAProxy 3.2.4.

docker run -d --name sp -e CONTAINERS=1 -e NETWORKS=1 \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -p 127.0.0.1:12375:2375 tecnativa/docker-socket-proxy:v0.4.2
sleep 2
curl -sN -o /dev/null http://127.0.0.1:12375/events &   # the long-lived watch
sleep 3
time docker stop -t 60 sp
scenario result
nothing connected 1s, exit 0
idle keep-alive, request already completed 1s, exit 0
/events open · -t 60 61s, exit 137
/events open · -t 700 700s, exit 137
/events client process killed first 11s, exit 0
/events client container removed first 1s, exit 0

The -t 700 row is the one worth noting: 700 was chosen to exceed the timeout client 10m in the defaults, to test whether that timeout would eventually end the session. It does not — HAProxy went straight past 600s.

The last two rows show the hang needs the session to be still established when SIGTERM lands. That makes it a race, which is presumably why it is not reported more often; a client that has already gone lets the proxy exit in about a second.

Impact

Anything that stops the proxy while its client is still connected pays the full timeout. In our case a monthly maintenance window held a production outage open for ten minutes — the whole docker compose stop -t 600 budget — for a stateless proxy with nothing worth waiting for.

Suggested fix

hard-stop-after in the global section of the template. Verified locally by passing a fragment as a second -f file:

global
    hard-stop-after 5s
[WARNING] (12) : Proxy dockerfrontend hard-stopped (1 remaining conns will be closed).
[WARNING] (1)  : All workers exited. Exiting... (0)

Result: 5s, exit 0, and the proxy serves normally in the meantime (200 on /containers/json). It bounds only the shutdown, so #62's behaviour during normal operation is untouched — the event stream still runs uncut for as long as the container is up.

Something in the 5–10s range as a default would fit Docker's own 10s stop grace period. Making it configurable via an env var, like the endpoint permissions, would suit anyone proxying a stream that genuinely needs longer to drain.

Happy to open a PR if the approach looks right.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the HAProxy global section of the template and reproduce the open /events shutdown with the provided docker run, curl, and docker stop commands. Evaluate the proposed hard-stop-after default or configuration path, then verify that shutdown is bounded while the /events stream remains available during normal operation.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
57/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.