hardbyte / hardbyte/postgresql-job-queue-benchmarking

Sweep follow-ups deferred from #26 / #27

Open
#28 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
2
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Tracking the follow-ups deferred from the 2026-05-09 sweep writeup ([#26](https://github.com/hardbyte/postgresql-job-queue-benchmarking/pull/26)) and the harness/adapter sweep-learnings PR ([#27](https://github.com/hardbyte/postgresql-job-queue-benchmarking/pull/27)). One issue rather than seven; tick boxes as items land.

## Bench-repo follow-ups

### Mixed-queue isolation plot (the deferred 6th plot)

- [ ] Per-queue counters in \`awa-bench/src/long_horizon.rs\` and \`pgque-bench/main.py\` — emit \`subject_kind=\"queue\"\`, \`subject=\` rows in addition to the existing \`adapter\` rollup. ~80 lines per adapter.
- [ ] \`scripts/render_queue_isolation.py\` (or extension to \`render_plots.py\`) producing two-panel plot:
- **Panel A — fair share**: per-queue completion rate over time, four lines per system, side-by-side awa | pgque.
- **Panel B — isolation under asymmetric load**: same plot with \`BENCH_QUEUE_PATTERN=80,10,5,5\` (or equivalent) — answers the actual operations question \"if one queue gets bursty do the others suffer?\"
- [ ] Rerun the two existing cells (\`mixed_queue_awa\`, \`mixed_queue_pgque\`) on the NixOS workstation; ~12 min wall.
- [ ] Optional: add a \`mixed_queue_asymmetric\` scenario to the harness for Panel B.

### Cross-adapter chaos coverage gap

- [ ] DLQ surface coverage for the six adapters that currently don't exercise it (oban, river, pgmq, pgboss, absurd, procrastinate). Each adapter has *some* notion of failed-job tracking — even \`(discarded)\` for oban — but the bench long_horizon worker only emits successful completions, so retry/DLQ throughput numbers don't exist for these systems.

### Harness improvements that didn't make it into PR #27

- [ ] Pre-flight reconnect-survival smoke per adapter at startup: open and close a backend, confirm the adapter recovers. Catches \"adapter has no reconnect\" *before* the four-minute chaos cell, in one second. PR #27 lifts the reconnect helpers themselves; this is the test harness around them.
- [ ] \`subject_kind=\"shutdown_phase\"\` row emission from the orchestrator when SIGTERM is sent, exit observed, and (if escalated) SIGKILL fired. Currently the per-replica breadcrumb in PR #27 lands in stderr / \`logs/.log\`; promoting it to \`raw.csv\` makes shutdown timing analyzable across cells. Needs schema discussion (probably reuses existing columns with a typed metric name).
- [ ] Manifest-level \`bulk_default\` field that \`adapter.json\` declares and the harness lints at startup. Today \`PRODUCER_BATCH_MAX\` defaults are scattered across each adapter and we re-discover the gimping case-by-case (oban + river default-1 was the latest). A manifest field plus a startup assertion stops the next adapter from shipping with \`default=1\`.

### Asymmetric-load mixed_queue follow-up

- [ ] Once Panel B above lands, file an awa repo issue if the per-queue isolation looks soft — awa's current dispatcher distributes claims across all subscribed queues without per-queue rate fairness guarantees. The bench result will tell us whether that's actually a problem.

## awa-side follow-ups (track in awa repo)

- [ ] Bench-cell aging not firing — \`aged_completion_rate=0\` in \`starvation_awa_60min\`. The mechanism works ([awa#249](https://github.com/hardbyte/awa/pull/249) confirms at 100 ms interval); the gap is the bench cell's load shape doesn't accumulate 60 s of wait time on the priority-4 lane. Either lower \`PRIORITY_AGING_MS\` for the bench cell or raise producer rate above the priority-1 lane's worker capacity.
- [ ] Rescue-cost reproducibility — the 33 % drop the 2026-05-08 rescue probe attributed at 1×256 doesn't reproduce in the 2026-05-09 sweep (-7.5 % / -1.3 % / -4.0 % at W=64 / 128 / 256). Settle whether rescue is firing at all on this workload shape.
- [ ] Bloat-resistance experiments under \`idle_in_tx\` and \`active_readers\` — awa drops to 75 % / 71 %, pgque holds 100 %. Reproduce locally with wait-event capture, identify whether the bottleneck is per-claim row locks or autovacuum starvation on \`lease_claims_*\`.
- [ ] Completion-batch experiment — prototype \`WORKER_COMPLETION_BATCH_MAX\` behind a feature flag, rerun the A.5-shape cells at W=64/128/256 to measure lift. Pgque-inspired; expected ceiling lift to 20 k+ jobs/s if the completion-side batch is real.

## Out of scope / parking lot

- Producer-rate sweeps per system (would multiply cell count by 3-4×).
- Per-system worker counts above 256.
- More aggressive chaos producer rates above 600 jobs/s.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.