Automated adversarial upgrade rehearsal (mixed-version, live traffic) in CI
- Dominant language
- Rust
- Stars
- 32
- Forks
- 5
- Avg merge
- 15h 21m
- Merged PRs (30d)
- 21
Description
## Why
ADR-041 requires an adversarial mixed-version rehearsal for migrations that change hot-path storage or an on-disk representation. The v043 ring-cursor migration therefore needs an automated test using the released N-1 artifact and the current build before a 0.7 tag is published.
The 2026-07-12 manual rehearsal did **not** test a mixed fleet: the old process was hard-killed, migration ran without workers or a producer, and the new fleet started afterward. It proved transactional migration, hard-kill recovery, scheduled work, exact reconciliation across 50,772 jobs, and the destructive #392 failure mode. It did not prove migration under live old workers, concurrent 0.6.2/0.7 operation, authority flip, or the returning-old-binary fence.
The strengthened 0.5-to-0.6 storage-transition rehearsals are also separate evidence. They prove canonical-to-queue-storage transition behavior, not the 0.6.2-to-0.7 schema and authority protocol.
## Required artifacts
- Released N-1: `awa-pg==0.6.2` and/or `ghcr.io/hardbyte/awa:0.6.2`.
- Current: the exact source SHA under test, built once and reused by every cell.
- Postgres version matching the release CI baseline.
Source builds of the 0.6 branch do not satisfy the released-artifact requirement.
## Required cells
### 1. Binary-first
Run current binaries against the finalized v040 schema before migration, keep traffic live, apply v041-v043, and continue through the mixed-fleet and flip phases.
### 2. Migrate-first
Apply v041-v043 while only released 0.6.2 runtimes are live, continue claiming on v043 in `columns` authority, then roll runtimes individually to the current build.
During the all-0.6.2-on-v043 interval, create deadline-bearing compact claims and assert the documented boundary: crash and heartbeat rescue continue; deadline-overrun batch claims may remain pending until the first current maintenance runtime starts, then each is rescued exactly once.
### 3. Overlap
Run migration while released 0.6.2 and current workers are both live. Assert both versions claim and complete work concurrently after migration and before flip.
## Workload
Keep continuous enqueue active across every phase and bank a backlog before migration. Include normal jobs, failures with retry, exhausted retries entering DLQ, future-scheduled jobs, cron jobs, deadline-bearing jobs, callback waits, cancellations, and multi-second jobs so work remains in flight.
Hard-kill at least one worker mid-job during the mixed window. Its jobs must be rescued at the expected attempt with exactly one terminal outcome.
## Flip and fence
Immediately before flip, force a final released-0.6.2 rotation that advances only the compatibility columns. The flip must lock the three cursor singletons, treat the columns as source of truth, reconcile every missing ledger generation, verify all three ledger cursors, install the fence, and promote authority atomically.
After flip, restart the released 0.6.2 artifact. It must fail loudly before it can rotate, misroute, or prune; the compatibility cursor and slot metadata must remain poisoned.
## Reconciliation
Stop the producer, drain, and reconcile from the discrete storage tables rather than `awa.jobs` (#422). Require:
- every accepted job has exactly one designed terminal outcome;
- killed jobs are recovered at the expected attempt;
- scheduled and cron work fired;
- retries and DLQ outcomes match their definitions;
- no deferred or in-flight work remains stuck;
- terminal rollups plus unfolded deltas are exact; and
- rotation ledgers and delta landing tables remain bounded after their maintenance window.
## Automation
Provide one deterministic local entry point and a GitHub Actions job. Run the full matrix nightly and make a release-candidate variant a required **pre-tag** check. A tag-only check is insufficient because the existing tag workflow begins publishing artifacts immediately.
Log the released artifact versions, current SHA, phase transitions, census watermarks, flip status, and final reconciliation in machine-readable artifacts. Failures must retain Postgres and worker logs.
## Relationship to existing coverage
- `scripts/compat-matrix.sh` is the static binary/schema leg. Add a pinned 0.6.2 supported-contract cell there; keep older artifacts only as additional regression or negative-policy legs.
- This issue is the dynamic live-traffic leg.
- #425 supplies the expand/flip/fence mechanism; #426 supplies fail-safe migration and preflight behavior.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with scripts/compat-matrix.sh and the existing tag workflow, then read the expand/flip/fence and fail-safe behavior supplied by #425 and #426. Add the dynamic live-traffic rehearsal with a deterministic local entry point and GitHub Actions coverage. Done means the required cells, workload, fence checks, reconciliation, machine-readable artifacts, retained logs, nightly matrix, and required pre-tag variant all pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, postgres, python, rust, shell
- Domain
- backend, ci-cd, databases, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100