hardbyte / hardbyte/awa

Backpressure policy: queue depth limits and producer-side flow control

Open
#341 0 comments 0 reactions 0 assignees View on GitHub
feature operational
Dominant language
Rust
Stars
32
Forks
5
Avg merge
15h 21m
Merged PRs (30d)
21

Description

## Problem

Nothing in Awa stops producers from outrunning completion. The evidence is already in the tracker:

- The #246 single-shard worst case: ~9,982/s enqueue against ~4,016/s completion — depth peaked at 1.26M with 33s e2e p99 in a 3-minute window.
- The #169 long-horizon runs: enqueue holds 800/s while pinned-horizon completion sags, and depth grows to six figures with no signal back to producers.

The COPY producer path (ADR-008) makes it very easy to enqueue far faster than any worker fleet can drain. Operators currently discover this as latency, not as an explicit signal.

## Building blocks that already exist

- `queue_counts_fast` (#289) — cheap, index-only depth probe designed for high-cadence polling.
- Lane-head cursor signals (#330) — depth derivable without exact scans.

## Design questions

1. **Policy surface**: per-queue `max_depth` with a configurable action — reject enqueue (error), soft signal (enqueue succeeds, returns depth/warning), or shed (drop oldest deferred work — probably out of scope).
2. **Transactional-enqueue tension**: rejecting an enqueue inside a user transaction (ADR-006) turns a queue-health condition into a business-transaction failure. A soft-signal default with opt-in hard rejection is probably the right shape; a hard reject must be documented as a deliberate coupling.
3. **Producer-side helper**: a depth-aware batch producer (pause/slow when depth > target) would capture most of the value without changing enqueue semantics at all — the benchmark harness already implements exactly this pattern externally.
4. **Where the check runs**: depth probe per enqueue batch (cheap via #289) vs. cached per-producer with a TTL.

Refs: ADR-006, ADR-008, #246, #169, #289, #330.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.