buzz-relay: git conformance probe hangs startup forever when a backend connection stalls (no request timeout, no deadline)
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Symptom
Fresh self-hosted deployment from the `deploy/compose` bundle (image `ghcr.io/block/buzz:main`, the bundle's pinned MinIO `RELEASE.2025-09-07T16-13-09Z`). The relay boots, starts the A3 conformance probe, logs two transport drops, and then goes silent **forever**:
```
14:34:29 INFO running git object-store conformance probe (A3 gate) race_width=32 race_rounds=3
14:34:30 WARN transport drop (pre-classification: socket/send failure) phase=if_match_race round=0 racer=14
14:34:33 WARN transport drop (pre-classification: socket/send failure) phase=if_none_match_race round=0 racer=6
(no further log output — observed for 10+ minutes before intervention)
```
The probe runs before the health/main listeners bind, so:
- `/_readiness` and the relay port refuse connections,
- the container healthcheck loops `unhealthy` indefinitely,
- nothing restarts it — the process is alive, just pending on a future that will never resolve.
Net effect: the documented self-hosted quickstart bricks whenever a single backend connection stalls, with no error, no timeout, and no diagnostic beyond silence. `BUZZ_GIT_CONFORMANCE_PROBE=false` is the only way out, and nothing tells the operator that.
## Root cause
Three things compound:
1. The shared `rust-s3` bucket client (`GitStore::new` → `Bucket::new`) sets **no request timeout** — a connection that is accepted but never answered pends indefinitely.
2. Both race phases collect racers with `futures_util::future::join_all`, which waits for **every** racer — one stalled socket parks the whole phase.
3. `run_conformance_probe` has no overall deadline, and `main.rs` awaits it before binding any listener.
The probe's existing drop-and-floor classification correctly handles racers that *fail* with `S3Error::{Reqwest, Http, Io}` — but a stalled connection never produces an error to classify. Under 32-way concurrent PUTs to the same key, MinIO occasionally does exactly this (note the adjacent socket/send failures in the same rounds).
## Deterministic repro (no MinIO required)
Point a `GitStore` at a TCP listener that accepts connections and never responds, then call `run_conformance_probe` — it never returns. The linked PR adds exactly this as a regression test ("tarpit").
## Proposed fix
See the linked PR: per-operation timeout (racer elapse = transport drop under the existing drop-and-floor rule, so A3 semantics are unchanged) plus a whole-probe deadline that fails closed with phase `deadline` and actionable guidance. Both env-overridable (`BUZZ_GIT_PROBE_OP_TIMEOUT_SECS`, `BUZZ_GIT_PROBE_DEADLINE_SECS`, matching the existing `BUZZ_GIT_PROBE_WRITERS`/`_ROUNDS` naming).
Related but distinct: #2470 is the probe *failing* on GCS (a classified, visible failure) and #2511 improves failure messaging + adds a Helm disable toggle — both assume the probe produces an outcome. This issue is the case where it produces nothing at all.
## Environment
- Relay: `ghcr.io/block/buzz:main` (2026-07-24), single-node `deploy/compose` on Docker 29.x
- Object store: MinIO `RELEASE.2025-09-07T16-13-09Z` (the bundle's pinned image), same host
- Probe config: defaults (width 32, rounds 3)
Contributor guide
Assessment
This issue has not been assessed yet.