hardbyte / hardbyte/awa

Automated adversarial upgrade rehearsal (mixed-version, live traffic) in CI

Open
#427 7 comments 0 reactions 0 assignees View on GitHub
correctness operational
Dominant language
Rust
Stars
32
Forks
5
Avg merge
15h 21m
Merged PRs (30d)
21

Description

## Why

ADR-041 requires an adversarial mixed-version rehearsal for migrations that change hot-path storage or an on-disk representation. The v043 ring-cursor migration therefore needs an automated test using the released N-1 artifact and the current build before a 0.7 tag is published.

The 2026-07-12 manual rehearsal did **not** test a mixed fleet: the old process was hard-killed, migration ran without workers or a producer, and the new fleet started afterward. It proved transactional migration, hard-kill recovery, scheduled work, exact reconciliation across 50,772 jobs, and the destructive #392 failure mode. It did not prove migration under live old workers, concurrent 0.6.2/0.7 operation, authority flip, or the returning-old-binary fence.

The strengthened 0.5-to-0.6 storage-transition rehearsals are also separate evidence. They prove canonical-to-queue-storage transition behavior, not the 0.6.2-to-0.7 schema and authority protocol.

## Required artifacts

- Released N-1: `awa-pg==0.6.2` and/or `ghcr.io/hardbyte/awa:0.6.2`.
- Current: the exact source SHA under test, built once and reused by every cell.
- Postgres version matching the release CI baseline.

Source builds of the 0.6 branch do not satisfy the released-artifact requirement.

## Required cells

### 1. Binary-first

Run current binaries against the finalized v040 schema before migration, keep traffic live, apply v041-v043, and continue through the mixed-fleet and flip phases.

### 2. Migrate-first

Apply v041-v043 while only released 0.6.2 runtimes are live, continue claiming on v043 in `columns` authority, then roll runtimes individually to the current build.

During the all-0.6.2-on-v043 interval, create deadline-bearing compact claims and assert the documented boundary: crash and heartbeat rescue continue; deadline-overrun batch claims may remain pending until the first current maintenance runtime starts, then each is rescued exactly once.

### 3. Overlap

Run migration while released 0.6.2 and current workers are both live. Assert both versions claim and complete work concurrently after migration and before flip.

## Workload

Keep continuous enqueue active across every phase and bank a backlog before migration. Include normal jobs, failures with retry, exhausted retries entering DLQ, future-scheduled jobs, cron jobs, deadline-bearing jobs, callback waits, cancellations, and multi-second jobs so work remains in flight.

Hard-kill at least one worker mid-job during the mixed window. Its jobs must be rescued at the expected attempt with exactly one terminal outcome.

## Flip and fence

Immediately before flip, force a final released-0.6.2 rotation that advances only the compatibility columns. The flip must lock the three cursor singletons, treat the columns as source of truth, reconcile every missing ledger generation, verify all three ledger cursors, install the fence, and promote authority atomically.

After flip, restart the released 0.6.2 artifact. It must fail loudly before it can rotate, misroute, or prune; the compatibility cursor and slot metadata must remain poisoned.

## Reconciliation

Stop the producer, drain, and reconcile from the discrete storage tables rather than `awa.jobs` (#422). Require:

- every accepted job has exactly one designed terminal outcome;
- killed jobs are recovered at the expected attempt;
- scheduled and cron work fired;
- retries and DLQ outcomes match their definitions;
- no deferred or in-flight work remains stuck;
- terminal rollups plus unfolded deltas are exact; and
- rotation ledgers and delta landing tables remain bounded after their maintenance window.

## Automation

Provide one deterministic local entry point and a GitHub Actions job. Run the full matrix nightly and make a release-candidate variant a required **pre-tag** check. A tag-only check is insufficient because the existing tag workflow begins publishing artifacts immediately.

Log the released artifact versions, current SHA, phase transitions, census watermarks, flip status, and final reconciliation in machine-readable artifacts. Failures must retain Postgres and worker logs.

## Relationship to existing coverage

- `scripts/compat-matrix.sh` is the static binary/schema leg. Add a pinned 0.6.2 supported-contract cell there; keep older artifacts only as additional regression or negative-policy legs.
- This issue is the dynamic live-traffic leg.
- #425 supplies the expand/flip/fence mechanism; #426 supplies fail-safe migration and preflight behavior.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with scripts/compat-matrix.sh and the existing tag workflow, then read the expand/flip/fence and fail-safe behavior supplied by #425 and #426. Add the dynamic live-traffic rehearsal with a deterministic local entry point and GitHub Actions coverage. Done means the required cells, workload, fence checks, reconciliation, machine-readable artifacts, retained logs, nightly matrix, and required pre-tag variant all pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, postgres, python, rust, shell
Domain
backend, ci-cd, databases, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.