Rust tests CI job takes ~14 minutes on main (37 minutes unsharded on release branches)

## Summary

The `Rust tests` matrix is the critical path of `ci.yml` on `main`: the slowest shard (`rest`, pg18) takes 13.6 minutes wall-clock, so the whole run waits ~14 minutes on it even though the sum of actual test time in that shard is ~10 minutes and most of the other shards finish in 4–7 minutes. On the release branches, which still run the pre-#335 unsharded job, the same tests take 36.7 minutes.

The dominant cost is not any individual assertion. It is that many integration tests rebuild the whole schema per test: `DROP SCHEMA awa CASCADE` followed by `migrations::run()` replays all 46 migrations (several of which reinstall the queue-storage substrate), which costs 7–9 s per test on a GitHub-hosted runner, and the binaries that do it serialise their tests behind a one-permit semaphore so the cost is paid strictly sequentially.

## Measurements

### `main` run [34793146226](https://github.com/hardbyte/awa/actions/runs/34793146226) (d6ac0334, push)

Job wall-clock (`startedAt` → `completedAt`) and the `Run test shard` step:

| Shard | pg17 job / step | pg18 job / step |
| --- | --- | --- |
| rest | 10.8 m / 10.2 m | **13.6 m / 13.1 m** |
| heavy | 7.1 m / 6.5 m | 7.7 m / 7.1 m |
| migrations-4 | 6.1 m / 5.5 m | 6.4 m / 5.9 m |
| migrations-2 | 6.2 m / 5.7 m | 5.3 m / 4.4 m |
| migrations-3 | 4.7 m / 3.9 m | 4.7 m / 4.1 m |
| migrations-1 | 4.0 m / 3.5 m | 3.9 m / 3.1 m |
| queue_storage leg | 1.1 m / 0.5 m | 1.0 m / 0.6 m |

Fixed overhead per job is ~0.5 m (`Initialize containers` 0.3–0.5 m, `rust-cache` restore ~0.2 m). The previous `main` run (34648903388) has the same shape: rest pg18 14.2 m, rest pg17 9.0 m, heavy 7.8–8.7 m, migrations 3.6–7.3 m.

### `rest` shard, pg18 (13.1 m step)

From the log timestamps: ~1.5 m compiling the `awa` test binaries, ~1.2 m compiling the non-`awa` workspace crates for `cargo test --workspace --exclude awa`, ~0.7 m compiling doctests, and 618 s (10.3 m) of test-harness time across 75 binaries. The top binaries by harness time (`finished in`):

| Binary | Tests | pg18 | pg17 |
| --- | --- | --- | --- |
| `awa/tests/enqueue_spec_test.rs` | 21 | **187 s** | 134 s |
| `awa/tests/cron_reconciliation_test.rs` | 28 (+1 ignored) | **110 s** | 81 s |
| `awa-worker` unit tests | 49 | 51 s | 37 s |
| `awa-cli/tests/storage_finalize_cli_test.rs` | 4 | 45 s | 33 s |
| `awa-cli/tests/migrate_cli_test.rs` | 8 | 30 s | 21 s |
| `awa-model/tests/quiesced_transition_test.rs` | 7 | 25 s | 18 s |
| `awa/tests/health_endpoint_test.rs` | 6 | 16 s | 13 s |
| everything else (68 binaries) | | < 12 s each | |

`enqueue_spec_test` runs its 21 tests one at a time behind a semaphore; each test drops the schema and replays the migration chain (~9 s), then runs a sub-second scenario. `cron_reconciliation_test` uses `#[sqlx::test]` (a fresh database per test) and then runs the full chain in each one behind a mutex.

### `heavy` shard, pg18 (7.1 m step)

| Binary | Tests | Harness time |
| --- | --- | --- |
| `awa/tests/lifecycle_hook_test.rs` | 21 | **180 s** (one test every ~8.5 s, serialised) |
| `awa/tests/queue_storage_runtime_test.rs` | 133 | 139 s |
| `awa/tests/cel_callback_test.rs` | 19 | 41 s |
| `awa/tests/executor_guard_test.rs` | 4 | 33 s (`test_deadline_rescue_signals_cancellation` ≈ 30 s) |
| `awa/tests/external_wait_test.rs` | 38 | 5 s |

`lifecycle_hook_test` has the same shape as `enqueue_spec_test`: drop schema, replay chain, install queue storage, run a short scenario — 21 times, serialised.

### `migrations-*` shards (3.1–5.9 m step each)

Each shard runs 17 `migration_test` tests under `cargo nextest --partition`, one process per test, 7–40 s per test (`test_v011_reseeds_singleton_when_upgrading_from_v010` 40 s, `test_v031_backfills_queue_storage_failed_done_metric_index` 38 s, `test_migration_sql_matches_run` 34 s, `test_v042_refreshes_compact_deadline_cursors_and_index` 30 s). Every test starts by dropping the schema and running the chain to head, and some then rewind and replay again. The partitions are uneven (3.1 m vs 5.9 m) because nextest partitions by count, not by cost.

### Unsharded job on `release/0.6.6`, run [34793883671](https://github.com/hardbyte/awa/actions/runs/34793883671) (e8591ffd)

Single `Rust tests` job: **36.7 minutes** (00:51:23 → 01:28:03). Compile 3.0 m; test-harness total 1950 s. `migration_test` alone: **1346 s** (22.4 m) for 44 tests, serialised by the file's mutex — 32 of them trip the harness's "has been running for over 60 seconds" warning because they queue behind the mutex. Next: `enqueue_spec_test` 134 s, `lifecycle_hook_test` 115 s, `queue_storage_runtime_test` 106 s.

## Root cause

A fresh head schema is built from scratch, per test, by replaying the full migration chain, and the binaries that do this serialise their tests so the replays add up linearly:

- `enqueue_spec_test` (21 tests), `lifecycle_hook_test` (21), `cron_reconciliation_test` (28): every test pays a full replay (`DROP SCHEMA` + `migrations::run`, or `#[sqlx::test]` fresh DB + `migrations::run`) for a scenario that itself takes well under a second. That is ~480 s of the ~1000 s of test time in `rest` + `heavy`.
- `migration_test` (68 tests): the same replay per test. About a fifth of those tests only need a head schema (gates, storage-transition state, exclusive-window preflight) and do not exercise the chain at all.
- The `rest` shard additionally carries the non-`awa` workspace crates (compile + `awa-worker`/`awa-cli`/`awa-model` tests + doctests, ~3.5 m), which no other shard shares.

Locally the pattern is the same at a smaller scale: `enqueue_spec_test` 67 s, `lifecycle_hook_test` 61 s, `cron_reconciliation_test` 36 s, each ~3 s of migration replay per test.

## Fix options

1. **Build the head schema once per process and clone it with `CREATE DATABASE … TEMPLATE`.** A clone takes ~100–150 ms (measured on Postgres 17 with the default `WAL_LOG` strategy; `FILE_COPY` forces checkpoints and is far slower). Each test then gets its own database, which also removes the need for the one-permit semaphores so the tests run in parallel. Applies directly to `enqueue_spec_test`, `lifecycle_hook_test`, `cron_reconciliation_test`, and the head-only `migration_test` cases. Coverage is unchanged: the chain is still exercised by the tests that are about the chain, and the fresh-install path is exercised by the template build itself.
2. **Rebalance shards.** Move the non-`awa` workspace crates and doctests out of `rest` into their own shard. Cheap, but on its own only trims ~3.5 m from `rest` and moves nothing structural.
3. **More nextest partitions for `migration_test`.** Brute force; spends runner minutes without reducing total work and leaves the release branches unchanged.
4. **Move long-horizon migration tests to the nightly suite.** Not recommended: the replay/atomicity/rerunnable tests are the guard for rolling-upgrade safety and should stay on the PR/merge gate.

**Recommendation:** 1 + 2. Expected result on `main`: `rest` and `heavy` drop to roughly 5–6 m each, the migrations shards shrink modestly, and the matrix's critical path falls from ~14 m to ~6–7 m. On the release branches the same change would take the unsharded job from ~37 m to roughly 15 m without touching the workflow; sharding it as `main` does would bring it in line.

Start with ci.yml and the setup patterns in awa/tests/enqueue_spec_test.rs, lifecycle_hook_test.rs, cron_reconciliation_test.rs, and migration_test. Run the affected cargo nextest suites to establish timings, then evaluate the proposed head-schema template cloning and shard rebalance. Done means coverage is preserved, tests can run without unnecessary serialization, and main and release CI approach the stated runtime targets.

Written by the indexing model from the issue text.