Deep fragmented backlog wedges drain at ~40% of ceiling with no recovery (single-generation claim filter)
- Dominant language
- Rust
- Stars
- 32
- Forks
- 5
- Avg merge
- 15h 21m
- Merged PRs (30d)
- 21
Description
**Measured magnitudes (orphan-swept cells, W=128, 4CPU-capped postgres):** at 8,000/s offered (past the ~6.5k drain knee), v0.6.0 drains 7,199/s (backlog →231k, p99 ~18.6s) and the 0.7 ledger branch drains 7,641/s (backlog →115k, p99 ~7.9s). Drain stays near the healthy ceiling; the dominant overload symptom is unbounded backlog + multi-second latency (the #341 backpressure gap). The single-generation yield ceiling below is real (deterministic seeder, identical on both versions) but requires deeper fragmentation than these shapes produce — it is a latent ceiling that mainly slows recovery from extreme backlogs.
Found while gating the 0.7-alpha perf stack; **reproduces on v0.6.0** — pre-existing, not caused by the 0.7 branches.
## Symptom
Once offered load exceeds the drain ceiling long enough for the ready backlog to fragment across many sealed ring generations, drain rate collapses and NEVER recovers, even though workers are idle-capable and the backlog is large:
- v0.6.0 @ 8,000/s offered, W=128 (4CPU-capped postgres): enqueue holds 7,998/s, backlog grows 77k→350k, late-window drain wedges at **3,358/s** (healthy ceiling is >5k at this shape).
- 0.7 ledger branch @ 5,000/s: same shape (its ceiling is lower, so it enters the regime at lower offered load; bisect evidence on PR #415).
## Mechanism (verified with a deterministic seeder + A/B on both versions)
`claim_ready_runtime`'s row-claim CTE filters to exactly ONE (ready_slot, ready_generation) per call — a claim cannot span generation boundaries. `v_claim_limit` (up to 512) is computed from the lane's whole outstanding, but the generation filter truncates the actual yield to the current generation's slice of the lane. With a backlog interleaved across N generations, yield-per-call ≈ slice size (measured: identical ~10/call on v0.6.0 and the ledger branch with burst-10 seeding across 15 generations). Under load, rotation stalls once all slots hold undrained rows (busy-check), pinning the fragmentation in place — the system stays in the low-yield regime until the entire backlog drains at the wedged rate.
There may additionally be a zero-yield call-storm component (measured mean yield ~10/call implies many empty routing/probe calls when slice arithmetic predicts ~500-row slices) — needs distribution-level measurement, not just pgss means.
## Why it matters
A transient overload burst (or a slow consumer deploy) that lets backlog fragment turns into a persistent ~60% throughput loss with no self-healing. Backpressure (#341) reduces entry probability but doesn't remove the cliff; adaptive claimers (#380) don't either (yield is per-call, not per-claimer).
## Candidate fix
Let one claim call span multiple CONTIGUOUS generations of the same lane up to `v_claim_limit` — raises the wedge floor to the true batch cap. This is a claim-CTE redesign (routing + attempt/receipt batch writes are keyed by (slot, generation)) and needs TLA+ model updates (AwaSegmentedStorage claim actions) — not a contained patch.
Evidence: bench repo `results/2026-07-11-awa-07-alpha-gate` + wedge cells (wedge_old/wedge_new); reproduction seeder described in PR #415's investigation thread.
Milestone: 0.7 (correctness/operational).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with claim_ready_runtime and its row-claim CTE, then read the reproduction seeder described in PR #415's investigation thread and the wedge cells in the referenced bench results. The work is done when claims can span contiguous generations up to v_claim_limit, the related TLA+ AwaSegmentedStorage claim actions are updated, and the deep-fragmentation wedge no longer collapses throughput.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgres, rust
- Domain
- backend-api-design, databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100