Routed reader statements are unbounded after acquire — mid-transaction blackhole hangs requests indefinitely (needs reader statement_timeout)
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
Once a routed read has acquired its reader connection, **nothing bounds any subsequent statement**. A replica path that goes dark *mid-transaction* (LB failover, security-group change, standby promotion, Aurora failover) converts a routed read into an indefinitely hung request — the writer-fallback code never regains control. Dawn reproduced a hang that ran to her 15s probe cap with no bound in sight (thread 65279d98 in #buzz-read-only-replica-usage, 2026-07-30).
Found during the #3268 activation review. **Not a #3268 merge blocker** (seams are dark until `BUZZ_REPLICA_READ_MAX_AGE_MS` is set, and the writer pool has the same missing timeout today), but it is an **activation gate**: #3268's one-snapshot-across-page+aux design holds reader transactions open longer than any pre-existing read, widening the exposure window.
## Evidence (all at `dd26caa9f`)
What IS bounded — Dawn measured the acquire path exhaustively with a silent TCP listener (accepts, never writes — the hard blackhole shape):
- Cold connect through acquire: 151–152ms `PoolTimedOut`, 3/3 (sqlx `inner.rs:252-255` wraps the whole acquire future in `rt::timeout`; connect is re-bounded inside at `:338/:350`).
- Idle-connection-gone-stale: 152ms (sqlx `test_before_acquire` defaults true, ping runs inside the acquire timeout).
What is NOT bounded — after `read_pool.acquire()` succeeds:
- `MIDTX_STATEMENT elapsed_ms=15002 result=STILL_HANGING_at_15s_cap (UNBOUNDED)` — BEGIN succeeded on a healthy replica, path blackholed, next statement hung past the probe cap.
- No bound exists anywhere (verified independently by Dawn and Eva at the SHA): `rg statement_timeout crates/ -g '*.rs'` → zero hits repo-wide; the reader pool builder (`lib.rs:723-731`) sets only max/min/acquire/lifetime/idle, no `after_connect`; no `tokio::time::timeout` wraps any routed read. Everything in `proved_reader` after acquire — Aurora probe (`:908`), `Transaction::begin` (`:909`), `observe_heartbeat` (`:955`) — and every page/aux statement on the held snapshot is unbounded.
## Fix (small)
Add `.after_connect` to `connect_read_pool` issuing `SET statement_timeout = '2s'` (value debatable; comfortably above any routed read, far below a hung request). The writer pool already uses the `after_connect` pattern for the floor guard (`lib.rs:686`), so this matches in-tree conventions. Reader-side is strictly safer than writer-side: replica sessions are read-only.
Bonus: this also bounds the recovery-conflict-cancellation family — a standby cancellation that never arrives is the same hang, and `statement_timeout` converts it into a `reader_validation_error` writer fallback.
Consider `idle_in_transaction_session_timeout` on the same `after_connect` while there (guards the snapshot-held-but-idle shape between page and aux queries).
Writer-pool `statement_timeout` is a separate, riskier discussion (writes mid-statement) — explicitly out of scope here.
## Acceptance
- Dawn's mid-tx blackhole probe bounded at the configured timeout with a `reader_validation_error` → writer fallback serving a complete, correct page (never partial).
- Live-rig repeat (Max/Wren topology): blackhole the reader path after a routed read acquires; request completes from the writer within timeout + one budget.
Refs: #3268 (activation gate item 5), #3643 (separate unfenced-FTS-pool family), #3622. Found by Dawn; provenance/negatives co-verified by Eva.
Contributor guide
Research direction
Start in lib.rs:723-731 at connect_read_pool and compare it with the writer pool's after_connect pattern at lib.rs:686. Trace the proved_reader path around lines 908-955, then run the mid-transaction blackhole probe described in the issue. Done means routed reader statements time out, return reader_validation_error, and serve a complete page through writer fallback; verify with the live-rig repeat.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100