block / block/buzz

buzz-push-gateway: blocking stdout logging can wedge the runtime (same class as relay fix #3256)

Open
#3,260 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

Split out of PR #3256 review (found by Dawn).

`crates/buzz-push-gateway/src/main.rs:22` still initializes logging with a plain blocking writer:

```rust
tracing_subscriber::fmt().json().with_env_filter(...).init();
```

It is deployed with a liveness probe (`deploy/charts/buzz-push-gateway/templates/deployment.yaml:45` — `timeoutSeconds: 3, failureThreshold: 3`). Same node, same containerd log copier, same failure mode as the relay: a stalled copier fills the ~64KB stdout pipe, the next `write(2)` blocks holding Rust's global stdout lock, the Tokio runtime parks, the probe times out, kubelet kills the pod. If the root cause is node-level disk-IO pressure, it hits both workloads simultaneously.

Fix: reuse the relay's approach from #3256 — `tracing_appender::non_blocking` lossy mode + `BoundedWorkerGuard` + drop-counter metric (`crates/buzz-relay/src/logging.rs`). Likely worth hoisting that module somewhere shared rather than copy-pasting.

Refs: PR #3256, RESEARCH/RELAY_STALL_LOCAL_REPRO_2026_07_27.md (Eva's nest).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.