Adaptive claimer autoscaling (experiment-gated, opt-in)
- Dominant language
- Rust
- Stars
- 32
- Forks
- 5
- Avg merge
- 15h 21m
- Merged PRs (30d)
- 21
Description
## Motivation
The single-shard/default-shape ceiling is real: the 2026-06-07 worst-case run accepted ~9,982 jobs/s enqueue but completed only ~4,016/s (depth peaked 1.26M) with one claimer. The remedy exists (`claimers`, `enqueue_shards`, `PartitionedQueue`) but requires hand-tuning. A bounded controller that adjusts per-queue claimers within `[1, max]` on a lane-lag signal could make the default shape absorb bursts without operators reading the throughput-lever docs first.
## Scope (experiment first — roadmap E4)
- Prototype: lag-signal controller vs static settings across steady / bursty / skewed load shapes.
- **Decision rule**: ship opt-in only if the controller shows no oscillation, no ordering-contract violation, and ≥1.5× default-shape completion on the burst shape. Otherwise close this issue with the data and ship documented tuning presets instead.
- If shipped: opt-in per queue (`claimers = auto(max)`), metrics for controller decisions, docs positioning it relative to `enqueue_shards` (which remains the semantic switch — this knob must never change ordering semantics).
## Acceptance
- E4 results recorded here with run ids (benchmark-repo convention).
- Either: opt-in feature merged with controller-behaviour tests, or issue closed with evidence + presets documented in `configuration.md`.
## References
Roadmap: `docs/0.7-roadmap.md` (WS-2, NEW-7; experiment E4). Evidence: #169 comment 2026-06-07 (`custom-20260607T104130Z-9ad735`). Related: ADR-025, ADR-031, #341.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with docs/0.7-roadmap.md and ADR-025/ADR-031, then review the benchmark-repo results and referenced run #169 comment (`custom-20260607T104130Z-9ad735`). Done means recording E4 run IDs and either merging the opt-in feature with controller-behaviour tests or documenting evidence and tuning presets in configuration.md.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, rust
- Domain
- backend, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100