buzz-push-gateway: blocking stdout logging can wedge the runtime (same class as relay fix #3256)
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
Split out of PR #3256 review (found by Dawn).
`crates/buzz-push-gateway/src/main.rs:22` still initializes logging with a plain blocking writer:
```rust
tracing_subscriber::fmt().json().with_env_filter(...).init();
```
It is deployed with a liveness probe (`deploy/charts/buzz-push-gateway/templates/deployment.yaml:45` — `timeoutSeconds: 3, failureThreshold: 3`). Same node, same containerd log copier, same failure mode as the relay: a stalled copier fills the ~64KB stdout pipe, the next `write(2)` blocks holding Rust's global stdout lock, the Tokio runtime parks, the probe times out, kubelet kills the pod. If the root cause is node-level disk-IO pressure, it hits both workloads simultaneously.
Fix: reuse the relay's approach from #3256 — `tracing_appender::non_blocking` lossy mode + `BoundedWorkerGuard` + drop-counter metric (`crates/buzz-relay/src/logging.rs`). Likely worth hoisting that module somewhere shared rather than copy-pasting.
Refs: PR #3256, RESEARCH/RELAY_STALL_LOCAL_REPRO_2026_07_27.md (Eva's nest).
Contributor guide
Assessment
This issue has not been assessed yet.