Backpressure policy: queue depth limits and producer-side flow control
- Dominant language
- Rust
- Stars
- 32
- Forks
- 5
- Avg merge
- 15h 21m
- Merged PRs (30d)
- 21
Description
## Problem
Nothing in Awa stops producers from outrunning completion. The evidence is already in the tracker:
- The #246 single-shard worst case: ~9,982/s enqueue against ~4,016/s completion — depth peaked at 1.26M with 33s e2e p99 in a 3-minute window.
- The #169 long-horizon runs: enqueue holds 800/s while pinned-horizon completion sags, and depth grows to six figures with no signal back to producers.
The COPY producer path (ADR-008) makes it very easy to enqueue far faster than any worker fleet can drain. Operators currently discover this as latency, not as an explicit signal.
## Building blocks that already exist
- `queue_counts_fast` (#289) — cheap, index-only depth probe designed for high-cadence polling.
- Lane-head cursor signals (#330) — depth derivable without exact scans.
## Design questions
1. **Policy surface**: per-queue `max_depth` with a configurable action — reject enqueue (error), soft signal (enqueue succeeds, returns depth/warning), or shed (drop oldest deferred work — probably out of scope).
2. **Transactional-enqueue tension**: rejecting an enqueue inside a user transaction (ADR-006) turns a queue-health condition into a business-transaction failure. A soft-signal default with opt-in hard rejection is probably the right shape; a hard reject must be documented as a deliberate coupling.
3. **Producer-side helper**: a depth-aware batch producer (pause/slow when depth > target) would capture most of the value without changing enqueue semantics at all — the benchmark harness already implements exactly this pattern externally.
4. **Where the check runs**: depth probe per enqueue batch (cheap via #289) vs. cached per-producer with a TTL.
Refs: ADR-006, ADR-008, #246, #169, #289, #330.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.