block / block/buzz

Routed reader statements are unbounded after acquire — mid-transaction blackhole hangs requests indefinitely (needs reader statement_timeout)

Open
#3,651 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

Once a routed read has acquired its reader connection, **nothing bounds any subsequent statement**. A replica path that goes dark *mid-transaction* (LB failover, security-group change, standby promotion, Aurora failover) converts a routed read into an indefinitely hung request — the writer-fallback code never regains control. Dawn reproduced a hang that ran to her 15s probe cap with no bound in sight (thread 65279d98 in #buzz-read-only-replica-usage, 2026-07-30).

Found during the #3268 activation review. **Not a #3268 merge blocker** (seams are dark until `BUZZ_REPLICA_READ_MAX_AGE_MS` is set, and the writer pool has the same missing timeout today), but it is an **activation gate**: #3268's one-snapshot-across-page+aux design holds reader transactions open longer than any pre-existing read, widening the exposure window.

## Evidence (all at `dd26caa9f`)

What IS bounded — Dawn measured the acquire path exhaustively with a silent TCP listener (accepts, never writes — the hard blackhole shape):
- Cold connect through acquire: 151–152ms `PoolTimedOut`, 3/3 (sqlx `inner.rs:252-255` wraps the whole acquire future in `rt::timeout`; connect is re-bounded inside at `:338/:350`).
- Idle-connection-gone-stale: 152ms (sqlx `test_before_acquire` defaults true, ping runs inside the acquire timeout).

What is NOT bounded — after `read_pool.acquire()` succeeds:
- `MIDTX_STATEMENT elapsed_ms=15002 result=STILL_HANGING_at_15s_cap (UNBOUNDED)` — BEGIN succeeded on a healthy replica, path blackholed, next statement hung past the probe cap.
- No bound exists anywhere (verified independently by Dawn and Eva at the SHA): `rg statement_timeout crates/ -g '*.rs'` → zero hits repo-wide; the reader pool builder (`lib.rs:723-731`) sets only max/min/acquire/lifetime/idle, no `after_connect`; no `tokio::time::timeout` wraps any routed read. Everything in `proved_reader` after acquire — Aurora probe (`:908`), `Transaction::begin` (`:909`), `observe_heartbeat` (`:955`) — and every page/aux statement on the held snapshot is unbounded.

## Fix (small)

Add `.after_connect` to `connect_read_pool` issuing `SET statement_timeout = '2s'` (value debatable; comfortably above any routed read, far below a hung request). The writer pool already uses the `after_connect` pattern for the floor guard (`lib.rs:686`), so this matches in-tree conventions. Reader-side is strictly safer than writer-side: replica sessions are read-only.

Bonus: this also bounds the recovery-conflict-cancellation family — a standby cancellation that never arrives is the same hang, and `statement_timeout` converts it into a `reader_validation_error` writer fallback.

Consider `idle_in_transaction_session_timeout` on the same `after_connect` while there (guards the snapshot-held-but-idle shape between page and aux queries).

Writer-pool `statement_timeout` is a separate, riskier discussion (writes mid-statement) — explicitly out of scope here.

## Acceptance

- Dawn's mid-tx blackhole probe bounded at the configured timeout with a `reader_validation_error` → writer fallback serving a complete, correct page (never partial).
- Live-rig repeat (Max/Wren topology): blackhole the reader path after a routed read acquires; request completes from the writer within timeout + one budget.

Refs: #3268 (activation gate item 5), #3643 (separate unfenced-FTS-pool family), #3622. Found by Dawn; provenance/negatives co-verified by Eva.

Contributor guide

Open the contributing guide

Research direction

Start in lib.rs:723-731 at connect_read_pool and compare it with the writer pool's after_connect pattern at lib.rs:686. Trace the proved_reader path around lines 908-955, then run the mid-transaction blackhole probe described in the issue. Done means routed reader statements time out, return reader_validation_error, and serve a complete page through writer fallback; verify with the live-rig repeat.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.