hardbyte / hardbyte/postgresql-job-queue-benchmarking

Sweep follow-ups deferred from #26 / #27

Offen
#28 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
HTML
Sterne
2
Forks
0
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

Tracking the follow-ups deferred from the 2026-05-09 sweep writeup ([#26](https://github.com/hardbyte/postgresql-job-queue-benchmarking/pull/26)) and the harness/adapter sweep-learnings PR ([#27](https://github.com/hardbyte/postgresql-job-queue-benchmarking/pull/27)). One issue rather than seven; tick boxes as items land.

## Bench-repo follow-ups

### Mixed-queue isolation plot (the deferred 6th plot)

- [ ] Per-queue counters in \`awa-bench/src/long_horizon.rs\` and \`pgque-bench/main.py\` — emit \`subject_kind=\"queue\"\`, \`subject=\` rows in addition to the existing \`adapter\` rollup. ~80 lines per adapter.
- [ ] \`scripts/render_queue_isolation.py\` (or extension to \`render_plots.py\`) producing two-panel plot:
- **Panel A — fair share**: per-queue completion rate over time, four lines per system, side-by-side awa | pgque.
- **Panel B — isolation under asymmetric load**: same plot with \`BENCH_QUEUE_PATTERN=80,10,5,5\` (or equivalent) — answers the actual operations question \"if one queue gets bursty do the others suffer?\"
- [ ] Rerun the two existing cells (\`mixed_queue_awa\`, \`mixed_queue_pgque\`) on the NixOS workstation; ~12 min wall.
- [ ] Optional: add a \`mixed_queue_asymmetric\` scenario to the harness for Panel B.

### Cross-adapter chaos coverage gap

- [ ] DLQ surface coverage for the six adapters that currently don't exercise it (oban, river, pgmq, pgboss, absurd, procrastinate). Each adapter has *some* notion of failed-job tracking — even \`(discarded)\` for oban — but the bench long_horizon worker only emits successful completions, so retry/DLQ throughput numbers don't exist for these systems.

### Harness improvements that didn't make it into PR #27

- [ ] Pre-flight reconnect-survival smoke per adapter at startup: open and close a backend, confirm the adapter recovers. Catches \"adapter has no reconnect\" *before* the four-minute chaos cell, in one second. PR #27 lifts the reconnect helpers themselves; this is the test harness around them.
- [ ] \`subject_kind=\"shutdown_phase\"\` row emission from the orchestrator when SIGTERM is sent, exit observed, and (if escalated) SIGKILL fired. Currently the per-replica breadcrumb in PR #27 lands in stderr / \`logs/.log\`; promoting it to \`raw.csv\` makes shutdown timing analyzable across cells. Needs schema discussion (probably reuses existing columns with a typed metric name).
- [ ] Manifest-level \`bulk_default\` field that \`adapter.json\` declares and the harness lints at startup. Today \`PRODUCER_BATCH_MAX\` defaults are scattered across each adapter and we re-discover the gimping case-by-case (oban + river default-1 was the latest). A manifest field plus a startup assertion stops the next adapter from shipping with \`default=1\`.

### Asymmetric-load mixed_queue follow-up

- [ ] Once Panel B above lands, file an awa repo issue if the per-queue isolation looks soft — awa's current dispatcher distributes claims across all subscribed queues without per-queue rate fairness guarantees. The bench result will tell us whether that's actually a problem.

## awa-side follow-ups (track in awa repo)

- [ ] Bench-cell aging not firing — \`aged_completion_rate=0\` in \`starvation_awa_60min\`. The mechanism works ([awa#249](https://github.com/hardbyte/awa/pull/249) confirms at 100 ms interval); the gap is the bench cell's load shape doesn't accumulate 60 s of wait time on the priority-4 lane. Either lower \`PRIORITY_AGING_MS\` for the bench cell or raise producer rate above the priority-1 lane's worker capacity.
- [ ] Rescue-cost reproducibility — the 33 % drop the 2026-05-08 rescue probe attributed at 1×256 doesn't reproduce in the 2026-05-09 sweep (-7.5 % / -1.3 % / -4.0 % at W=64 / 128 / 256). Settle whether rescue is firing at all on this workload shape.
- [ ] Bloat-resistance experiments under \`idle_in_tx\` and \`active_readers\` — awa drops to 75 % / 71 %, pgque holds 100 %. Reproduce locally with wait-event capture, identify whether the bottleneck is per-claim row locks or autovacuum starvation on \`lease_claims_*\`.
- [ ] Completion-batch experiment — prototype \`WORKER_COMPLETION_BATCH_MAX\` behind a feature flag, rerun the A.5-shape cells at W=64/128/256 to measure lift. Pgque-inspired; expected ceiling lift to 20 k+ jobs/s if the completion-side batch is real.

## Out of scope / parking lot

- Producer-rate sweeps per system (would multiply cell count by 3-4×).
- Per-system worker counts above 256.
- More aggressive chaos producer rates above 600 jobs/s.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Start by reading PRs #26 and #27, then choose one scoped follow-up and inspect the named entry points: awa-bench/src/long_horizon.rs, pgque-bench/main.py, scripts/render_queue_isolation.py, or render_plots.py. Run the relevant mixed_queue or starvation benchmark first. Done means the selected checkbox is implemented, its benchmark or output is reproducible, and any required result or schema is documented.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
nixos, postgresql, python, rust
Bereich
backend, databases, performance, testing, tooling
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Ruhig
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.