Firehose block stream can hang indefinitely with no idle/read timeout → subgraph silently stops indexing until restart

Summary

A subgraph fed by a firehose block stream can stop indexing indefinitely and silently if the firehose upstream keeps the HTTP/2 stream open but stops sending frames (no message, no error, no end-of-stream). The subgraph stays health: healthy, is not paused, sets no fatalError, emits zero logs, and its latestBlock is frozen. Only a manual restart / reassignment (which opens a fresh stream) recovers it.

The block-stream receive loop has no per-message idle/read timeout, and HTTP/2 keepalive pings are intentionally disabled, so there is nothing to detect or break a half-open / app-hung upstream.

Affected versions

Confirmed present and unchanged from v0.42.1 through v0.44.0 and current master. The relevant code has not changed since it was introduced.

Symptom (observed in production)

• graphman info --status: Paused: false, Health: healthy, fatalError: null.
• latestBlock frozen for hours; no forward progress.
• No log lines for the deployment anywhere (all pods) for the entire stall.
• Other subgraphs on the same node/chain keep indexing normally (node is fine).
• graphman reassign <hash> <node> (fresh runner → fresh stream) recovers it immediately.
• Correlated with the firehose's upstream block source being briefly unstable (e.g. a node restart), which can leave individual gRPC streams half-open.

Root cause

1. No idle timeout on the stream receive loop.
graph/src/blockchain/firehose_block_stream.rs (master: establishment timeout at L246, receive loop at L258):

let req = endpoint.clone().stream_blocks(request, &headers);
let result = tokio::time::timeout(Duration::from_secs(120), req).await; // guards ESTABLISHMENT only
match result {
    Ok(stream) => {
        let mut last_response_time = Instant::now();  // only feeds a metric
        for await response in stream {                // <-- no timeout / select! around each .next()
            ...
        }

• The tokio::time::timeout(120s, req) wraps only the initial stream_blocks establishment future, not the per-frame iteration.
• Inside for await response in stream there is no tokio::time::timeout, no select!, no idle deadline. last_response_time is only passed to a metric (observe_response) and never used to trip a timeout.
• If the upstream holds the HTTP/2 stream open and sends nothing, for await never yields, and the hang propagates cleanly up the whole consumer chain with no error and no log:
  • BufferedBlockStream (graph/src/blockchain/block_stream.rs:62) only pumps the stream into an mpsc channel — no timeout of its own.
  • The runner's block_stream.next().await (core/src/subgraph/runner/mod.rs:327) has no tokio::time::timeout and no select!.
  • The only wakeup for the suspended runner task is its Cancelable/CancelGuard, which is fired only by unassign/reassign. This is exactly why graphman reassign (fresh runner → fresh stream) is the only thing that recovers it.

2. HTTP/2 keepalive intentionally disabled (no liveness probe for a half-open stream).
graph/src/firehose/endpoints.rs (~L241–250 on master):

// Note: Do not set `http2_keep_alive_interval` or `http2_adaptive_window`, as these will
// send ping frames, and many cloud load balancers will drop connections that frequently
// send pings.
let endpoint = endpoint_builder
    .initial_connection_window_size(Some((1 << 31) - 1))
    .connect_timeout(Duration::from_secs(10))
    .tcp_keepalive(Some(Duration::from_secs(15)))
    // Timeout on each request, so the timeout to establish each 'Blocks' stream.
    .timeout(Duration::from_secs(120));

• HTTP/2 keepalive pings are deliberately off, so there is no application-level liveness probe.
• tcp_keepalive(15s) detects a fully dead peer (no TCP ACKs) but not a half-open / app-hung peer whose OS keeps the socket alive — the common case when the upstream is mid-restart behind a proxy/LB.
• connect_timeout and .timeout(120s) (per the code's own comment) bound only stream establishment, not mid-stream idle.

This is a known, deliberately worked-around limitation, not an oversight. The original comment on the connection-window tuning (PR #3818, 2022-08-08) states:

"We run multiple block streams on a same connection, and a problematic subgraph with a stalled block stream might consume the entire window capacity for its http2 stream and never release it. If there are enough stalled block streams to consume all the capacity on the http2 connection, then all subgraphs using this same http2 connection will stall."

The mitigation chosen at the time was to set the connection window to the maximum (so a few stalled streams don't starve others) — it makes the connection tolerate stalled streams but never recovers an individual stalled stream.

Why existing guards don't cover it

• connect_timeout / .timeout(120s) — establishment only (see comment in code).
• tcp_keepalive(15s) — dead peer only, not half-open.
• No GRAPH_* env var bounds the streaming-loop idle time. GRAPH_FIREHOSE_FETCH_BLOCK_TIMEOUT_SECS (firehose_block_fetch_timeout) only guards the reorg block-refetch RPC, not the main stream loop.
• GRAPH_KILL_IF_UNRESPONSIVE does not catch this: it is a tokio-runtime/threadpool contention watchdog (ping/pong over the whole runtime, node/src/launcher.rs). A runner suspended on an idle .await consumes no thread, so the watchdog stays satisfied. (The reporter of #4146 had GRAPH_KILL_IF_UNRESPONSIVE=true; it detected nothing.)

Related / prior art (none fixes this)

• #4146 "Indexing stucks until restart" (closed, 2022) — near-identical symptom (stops indexing, no errors, clear logs, different subgraph each time, only a full restart fixes it), but on the RPC poller and closed without a root cause. Symptomatic precedent.
• #3810 — added the firehose connect_timeout (establishment-only; explicitly not mid-stream).
• #3855 "firehose: Set a timeout for grpc requests" — added the .timeout(120s), motivated explicitly by "we might have seen requests hanging" — but it bounds only stream establishment.
• #3822 "firehose: Set tcp keepalive" ("to detect firehose restarts and such") — TCP-level only; misses a half-open / app-hung peer.
• #3818 — connection window max + the "do not set http2_keep_alive_interval" comment quoted above.
• #4190 "Store connection issue prevents subgraph indexing until graph-node is restarted" — same shape (silent stall until restart) on the Postgres writer path.

Note: maintainers have already reached for hang mitigations twice (#3855 request timeout, #3822 TCP keepalive), but both bound only connection establishment / dead-socket detection — neither recovers a mid-stream idle hang.

Proposed fix (either, or both)

1. Idle timeout on the receive loop — wrap each stream.next() in tokio::time::timeout(idle_deadline, …) (or a select! against a reset-on-message deadline) inside the loop; on elapse, drop the stream and reconnect with backoff (the reconnect path already exists). Make idle_deadline env-configurable (e.g. GRAPH_FIREHOSE_STREAM_IDLE_TIMEOUT_SECS), disabled by default to preserve current behavior.
2. Re-enable a conservative HTTP/2 keepalive — http2_keep_alive_interval + keep_alive_timeout, gated behind an env var so operators whose network path is not behind a ping-dropping LB (e.g. an in-cluster service) can opt in. This directly negates the failure mode the 2022 comment was avoiding, without forcing it on everyone.

Option 1 is topology-independent and recommended as the primary fix.

Reproduction

1. Point a subgraph at a firehose endpoint.
2. While it streams, make the firehose upstream hold the gRPC stream open but stop emitting frames (e.g. pause/stall the upstream block source, or interpose a proxy that stops forwarding frames without closing the stream).
3. Observe: the subgraph freezes with health: healthy, no logs, no fatalError, frozen latestBlock, indefinitely.
4. Restart/reassign → recovers immediately.

Environment

• graph-node v0.42.1 (also verified unchanged on master / v0.44.0).
• Ethereum mainnet subgraphs consuming a firehose block stream (StreamingFast firehose-ethereum), eth_call provider separate.

Start with graph/src/blockchain/firehose_block_stream.rs, especially the stream establishment and receive loop, then inspect graph/src/blockchain/block_stream.rs and core/src/subgraph/runner/mod.rs for how suspension propagates. Review graph/src/firehose/endpoints.rs for existing timeout and keepalive behavior. Done should mean an idle firehose stream is detected and recovered without manual reassignment, while normal streams continue indexing.