block / block/buzz

buzz-relay accepts connections but never responds when the disk is full (ENOSPC), and ignores SIGTERM

Open
#4,302 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

When the host filesystem is full (`ENOSPC`), `buzz-relay` stops serving but keeps its listeners open. It accepts TCP connections on every port and then never writes a response — including on the metrics port. It also does not terminate on `SIGTERM` and has to be `SIGKILL`ed.

The practical consequence is that a relay in this state looks healthy to anything that checks it. A TCP connect succeeds, so port checks and most uptime probes pass, while every client hangs until its own timeout.

## Observed behaviour

`buzz-relay` v0.2.0, debug build, macOS (aarch64-apple-darwin), running ~14h.

All three listeners accepted the connection and never replied:

```
:3000 -> http 000 in 5.008s (timeout, connection accepted)
:8080 -> http 000 in 5.008s (timeout, connection accepted)
:9102 -> http 000 in 5.010s (timeout, connection accepted)
```

`:9102` is the most diagnostic of the three — a plain `GET /` on the metrics endpoint should return immediately regardless of application state, and it did not.

The process had healthy-looking dependencies throughout: 5 established connections to Postgres and 3 to Redis, and both datastores were independently responsive (`PING` → `+PONG`, Postgres accepting connections).

The process was **not** spinning — roughly 6m13s of CPU across 14 hours. It was blocked, not busy. A `sample` of the process showed 11 threads, all in the tokio runtime (`tokio::runtime::scheduler::multi_thread::worker::run`, `tokio::runtime::task::harness::poll_future`).

`kill -TERM` had no effect after 5+ seconds; `kill -9` was required.

## Root cause

The host's data volume was at 100% (208 MiB free of 460 GiB). This was confirmed independently — a `cargo build` of this same binary failed at the link step on the same box with:

```
ld: write() failed, errno=28 (No space left on device)
```

After freeing disk space the binary rebuilt and linked cleanly in 45s, and the restarted relay served normally (`:3000` → 200 in 30ms, `:9102` → 200 in 10ms). No code changed between the wedged run and the healthy run.

## Expected

Under `ENOSPC` I would expect some subset of:

1. The metrics/health listener keeps answering, so the condition is externally observable.
2. Write failures surface as 5xx rather than an indefinite hang, so clients fail fast instead of blocking.
3. `SIGTERM` is honoured, so supervisors can cycle the process.
4. The `ENOSPC` is logged. Nothing in the relay's own output indicated disk exhaustion — the condition had to be inferred from an unrelated linker error.

## Actual

Listeners accept and hang indefinitely, `SIGTERM` is ignored, and nothing is logged.

## Impact

A relay in this state is invisible to health checking. Every liveness signal available externally — port open, process running, database connections established — reports healthy while the service answers nothing. Recovery requires noticing the symptom by hand and `SIGKILL`ing.

## Notes

Happy to supply the `sample` output if useful. I did not attempt a minimal reproduction, but filling the volume the relay writes to should reproduce it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.