Test harness: run the suite under both storage engines (engine-aware helpers + engine guard)
- Dominant language
- Rust
- Stars
- 32
- Forks
- 5
- Avg merge
- 15h 21m
- Merged PRs (30d)
- 21
Description
## Motivation
awa's caller-facing contract (enqueue / claim / complete / cancel / admin reads) is **engine-invariant** — it must behave identically whether the active engine is `canonical` or `queue_storage`. But the default integration harness forces `canonical` (`awa_testing::setup::reset_runtime_backend`), so the queue-storage *implementations* of that contract are only covered by the dedicated `queue_storage_runtime_test` suite, not by the broad integration suite.
This blind spot is how the queue-storage `cancel_by_unique_key` candidate query (`{schema}.leases.unique_key`, a column that does not exist on `leases`) shipped — every `cancel_by_unique_key` test ran under canonical, so the queue-storage SQL branch was dead code to CI. (Fixed in #359 / 0.6.0-rc.4.)
## Experiment result
Running `integration_test` under a queue-storage activation (`AWA_TEST_ENGINE=queue_storage`) produces **31 pass / 13 fail**, and all 13 failures are **test artifacts**, in three buckets:
- **Harness uses canonical-only raw SQL.** `TestClient::work_one_in_queue` / `work_one` hand-roll the claim→run→complete loop with raw `UPDATE awa.jobs`, and several tests manufacture preconditions via raw `UPDATE/INSERT awa.jobs`. The view's INSTEAD-OF trigger rejects these under queue_storage by design (`0A000`). The real executor uses the substrate (`runtime.store.*`), so these helpers never exercised the real engine path.
- **Admin-metadata cache assertions** (`queue_state_counts` / dirty-queue bookkeeping) — canonical-trigger-maintained; differ/no-op under queue_storage.
- **Test isolation** — `clean_queue`'s `DELETE FROM awa.jobs WHERE queue = $1` does not release canonical `job_unique_claims` rows under queue_storage, so claims accumulate across the run and later inserts collide on `idx_awa_jobs_unique`. Each affected test passes fresh + in isolation.
Net: **zero real queue-storage engine bugs** surfaced — but the harness can't currently run the broad suite under queue_storage.
## Proposed work
1. **Engine-aware harness helpers.** Make `work_one_in_queue` / `work_one` / `clean_queue` (and any other raw-`awa.jobs` test helper) route through the real runtime/substrate path when the active engine is `queue_storage`, instead of raw `UPDATE/INSERT/DELETE awa.jobs`. Cleanup must release `job_unique_claims` under both engines.
2. **Engine guard for canonical-specific tests.** Provide a one-line skip/guard (e.g. an `AWA_TEST_ENGINE`-aware helper) for tests that assert canonical-only mechanics (admin-metadata caches, `*_canonical_*` notification tests). These stay canonical-only by design.
3. **CI matrix dimension.** Once the suite is green under both engines, run the correctness suites twice (`engine: [canonical, queue_storage]`). Scope the second pass to correctness suites; exclude soak/bench/chaos (already engine-specific).
## Outcome
The engine-invariant contract is asserted under both engines in CI, closing the class of "queue-storage branch is dead code to CI" defects (the `leases.unique_key` family).
A spike branch with the `AWA_TEST_ENGINE` parameterization and the 31/44 triage exists locally and can seed the implementation.
---
## Also: forward-compatibility (old binary against the newest schema)
Related coverage gap on the same axis. A consumer running an **old** awa version (e.g. 0.5.7, canonical) often keeps running against a database that has already been migrated to the newest schema (V39) — the mixed-version window during a rolling upgrade, and any deployment where migrations run ahead of the worker roll.
Current coverage:
- The default harness always migrates to `CURRENT_VERSION` and runs canonical, so the whole canonical suite exercises **current** canonical code against V39.
- `migration_test.rs` covers fresh-install→V39, triple-run idempotency, V1→V39 data survival, legacy normalization, per-version backfills, and auto-finalize-on-fresh-install (+ its refusal guards).
- The **old 11-arg `insert_job_compat` positional signature** is explicitly called against V39 (current Rust passes 12 args), proving old-caller overload resolution still works — but only for the single-insert path.
Gap: **no test runs a genuinely old binary against a V39 database.** Everything compiles/imports from the current tree; the only mixed-fleet test uses the current Python binding and is `#[ignore]`/nightly-only. `sql_only_storage_upgrade_test.rs` drives the engine transition but never claims/completes a job, so it does not assert the canonical work surface post-migration.
Proposed: a version-pinned forward-compat test — enqueue → claim → complete with awa **0.5.7** (pinned dev-dependency, or `pip install awa==0.5.7` for the Python path) against a freshly-V39-migrated database. This is the executable form of the existing analytical sign-off (0.5.7 DB-surface audit + migration-safety audit) and the strongest guard for the rolling-upgrade window.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with integration_test, awa_testing::setup::reset_runtime_backend, and the TestClient helpers work_one_in_queue, work_one, and clean_queue. Run the suite with AWA_TEST_ENGINE=queue_storage and compare the 31/44 triage, then identify canonical-only assertions for guards. Done means correctness suites pass under both engines in CI, with the proposed 0.5.7 forward-compatibility coverage addressed or scoped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python, rust
- Domain
- ci-cd, databases, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100