buzz-relay accepts connections but never responds when the disk is full (ENOSPC), and ignores SIGTERM
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
When the host filesystem is full (`ENOSPC`), `buzz-relay` stops serving but keeps its listeners open. It accepts TCP connections on every port and then never writes a response — including on the metrics port. It also does not terminate on `SIGTERM` and has to be `SIGKILL`ed.
The practical consequence is that a relay in this state looks healthy to anything that checks it. A TCP connect succeeds, so port checks and most uptime probes pass, while every client hangs until its own timeout.
## Observed behaviour
`buzz-relay` v0.2.0, debug build, macOS (aarch64-apple-darwin), running ~14h.
All three listeners accepted the connection and never replied:
```
:3000 -> http 000 in 5.008s (timeout, connection accepted)
:8080 -> http 000 in 5.008s (timeout, connection accepted)
:9102 -> http 000 in 5.010s (timeout, connection accepted)
```
`:9102` is the most diagnostic of the three — a plain `GET /` on the metrics endpoint should return immediately regardless of application state, and it did not.
The process had healthy-looking dependencies throughout: 5 established connections to Postgres and 3 to Redis, and both datastores were independently responsive (`PING` → `+PONG`, Postgres accepting connections).
The process was **not** spinning — roughly 6m13s of CPU across 14 hours. It was blocked, not busy. A `sample` of the process showed 11 threads, all in the tokio runtime (`tokio::runtime::scheduler::multi_thread::worker::run`, `tokio::runtime::task::harness::poll_future`).
`kill -TERM` had no effect after 5+ seconds; `kill -9` was required.
## Root cause
The host's data volume was at 100% (208 MiB free of 460 GiB). This was confirmed independently — a `cargo build` of this same binary failed at the link step on the same box with:
```
ld: write() failed, errno=28 (No space left on device)
```
After freeing disk space the binary rebuilt and linked cleanly in 45s, and the restarted relay served normally (`:3000` → 200 in 30ms, `:9102` → 200 in 10ms). No code changed between the wedged run and the healthy run.
## Expected
Under `ENOSPC` I would expect some subset of:
1. The metrics/health listener keeps answering, so the condition is externally observable.
2. Write failures surface as 5xx rather than an indefinite hang, so clients fail fast instead of blocking.
3. `SIGTERM` is honoured, so supervisors can cycle the process.
4. The `ENOSPC` is logged. Nothing in the relay's own output indicated disk exhaustion — the condition had to be inferred from an unrelated linker error.
## Actual
Listeners accept and hang indefinitely, `SIGTERM` is ignored, and nothing is logged.
## Impact
A relay in this state is invisible to health checking. Every liveness signal available externally — port open, process running, database connections established — reports healthy while the service answers nothing. Recovery requires noticing the symptom by hand and `SIGKILL`ing.
## Notes
Happy to supply the `sample` output if useful. I did not attempt a minimal reproduction, but filling the volume the relay writes to should reproduce it.
Contributor guide
Assessment
This issue has not been assessed yet.