Tecnativa / Tecnativa/docker-socket-proxy
Container never stops while an /events stream is open: soft-stop waits forever, no hard-stop-after
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.8k
- Forks
- 207
- PR merge metrics
- No merged PRs in 30d
Description
Summary
With a client holding /events open — Traefik's Docker provider does this permanently — the container never stops on its own. Not slowly: never. docker stop always ends in SIGKILL and exit 137 at whatever deadline it is given.
This is a consequence of the fix for #62, not a regression in it. The dedicated docker-events backend with timeout server 0 is right — an event stream should not be cut every ten minutes — but on SIGTERM HAProxy performs a soft stop and waits for established sessions to end, and that session has no end. Nothing bounds the wait, because hard-stop-after is not set.
HAProxy warns about this itself, at every boot:
[WARNING] (12) : missing timeouts for backend 'docker-events'.
| While not properly invalid, you will certainly encounter various problems
| with such a configuration. To fix this, please ensure that all following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Reproduction
v0.4.2, HAProxy 3.2.4.
docker run -d --name sp -e CONTAINERS=1 -e NETWORKS=1 \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-p 127.0.0.1:12375:2375 tecnativa/docker-socket-proxy:v0.4.2
sleep 2
curl -sN -o /dev/null http://127.0.0.1:12375/events & # the long-lived watch
sleep 3
time docker stop -t 60 sp
| scenario | result |
|---|---|
| nothing connected | 1s, exit 0 |
| idle keep-alive, request already completed | 1s, exit 0 |
/events open · -t 60 |
61s, exit 137 |
/events open · -t 700 |
700s, exit 137 |
/events client process killed first |
11s, exit 0 |
/events client container removed first |
1s, exit 0 |
The -t 700 row is the one worth noting: 700 was chosen to exceed the timeout client 10m in the defaults, to test whether that timeout would eventually end the session. It does not — HAProxy went straight past 600s.
The last two rows show the hang needs the session to be still established when SIGTERM lands. That makes it a race, which is presumably why it is not reported more often; a client that has already gone lets the proxy exit in about a second.
Impact
Anything that stops the proxy while its client is still connected pays the full timeout. In our case a monthly maintenance window held a production outage open for ten minutes — the whole docker compose stop -t 600 budget — for a stateless proxy with nothing worth waiting for.
Suggested fix
hard-stop-after in the global section of the template. Verified locally by passing a fragment as a second -f file:
global
hard-stop-after 5s
[WARNING] (12) : Proxy dockerfrontend hard-stopped (1 remaining conns will be closed).
[WARNING] (1) : All workers exited. Exiting... (0)
Result: 5s, exit 0, and the proxy serves normally in the meantime (200 on /containers/json). It bounds only the shutdown, so #62's behaviour during normal operation is untouched — the event stream still runs uncut for as long as the container is up.
Something in the 5–10s range as a default would fit Docker's own 10s stop grace period. Making it configurable via an env var, like the endpoint permissions, would suit anyone proxying a stream that genuinely needs longer to drain.
Happy to open a PR if the approach looks right.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the HAProxy global section of the template and reproduce the open /events shutdown with the provided docker run, curl, and docker stop commands. Evaluate the proposed hard-stop-after default or configuration path, then verify that shutdown is bounded while the /events stream remains available during normal operation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 57/100