buzz-relay lib tests are archived but not run in CI — 771 of 804 never execute, including the mesh and tunnel/lease suites
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
`buzz-relay`'s library tests are compiled in CI but almost none are executed. Of 804 tests in `cargo test -p buzz-relay --lib`, CI runs **12** — the `#[ignore]`d subset of `api::invites::tests`. All **771 non-ignored** tests, including the entire mesh, tunnel/lease, subscription, media-auth, and pubsub-fanout suites, execute nowhere in any workflow.
This looks unintentional rather than a deliberate policy: the backend-integration job already provisions Postgres, Redis, and MinIO, and the nextest archive is explicitly built with `-p buzz-relay --lib`. The filter that selects tests from that archive was written for the invite suite and appears never to have been broadened.
## Evidence
Every test invocation across `.github/workflows/`:
| Location | Runs |
|---|---|
| `ci.yml:129` | `just test-unit` → `buzz-core`, `buzz-auth`, `buzz-db --lib`, `buzz-conformance`, `buzz-push-gateway` (justfile:275). No `buzz-relay`. |
| `ci.yml:687` | archive, `-E '(package(buzz-db) and test(/relay_invite::tests/)) or (package(buzz-relay) and test(/api::invites::tests/))' --run-ignored ignored-only` |
| `ci.yml:699` | archive, `-E 'binary(e2e_event_reminder)' --run-ignored ignored-only` |
| `ci.yml:740-742` | `buzz-test-client` e2e (persona, nostr_interop, relay invite, nip43) |
| `ci.yml:970`, `ci.yml:1005` | `buzz-dev-mcp`, desktop Tauri (Windows) |
The archive is created at `ci.yml:351` with `-p buzz-db -p buzz-relay -p buzz-test-client --lib`, so the binaries exist; only those two `-E` filters ever select from it, and both pass `--run-ignored ignored-only`.
Reproduce the counts:
```bash
cargo test -p buzz-relay --lib -- --list | grep -c ': test$' # 804
cargo test -p buzz-relay --lib -- --list --ignored | grep -c ': test$' # 33
cargo test -p buzz-relay --lib -- --list --ignored | grep -c 'api::invites' # 12 ← what CI runs
```
## Why it matters
This is the reason #2458 survived in `main`. `api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo` fails roughly half the time on a machine with Redis — I measured 5/10 failures on `origin/main` @ 22be8bb3, and @tlongwell-block and two others reproduced it independently this week. A test with that failure rate cannot survive a CI that runs it. It survived because CI never has.
The same blind spot covers `tunnel::directory` (lease acquisition, fencing, TTL takeover) and `tunnel::reliable` (ownership routing, renewer loss signalling) — the coordination primitives underneath the mesh.
There is a second, related trap that @tlongwell-block flagged on #2458: 11 of these tests call `redis_directory_if_available()` and **return early, silently, and green** when Redis is unreachable (`tunnel/directory.rs` ×4, `tunnel/reliable.rs` ×4, `api/mesh_demo.rs` ×2, `handlers/event.rs` ×1). So even running them locally without infra proves nothing, and the output is indistinguishable from real coverage.
## Cost to fix
Approximately zero. The full non-ignored suite against local Postgres + Redis:
```
test result: ok. 771 passed; 0 failed; 33 ignored; 0 measured; 0 filtered out; finished in 2.22s
```
2.22 seconds, all green, in a job that already has the services running.
## Suggested fix
Add a step to `backend-integration` (which already has Postgres/Redis/MinIO up and the archive downloaded) running the `buzz-relay` lib suite from the existing archive.
Ordering matters: enabling this today makes CI flake about half the time on the #2458 test. #2745 fixes that flake — I isolated the mechanism to cancel-safety in `run_demo_echo` with a 2×2 matrix at N=10 per cell ([detail](https://github.com/block/buzz/issues/2458#issuecomment-5112532129)) — so the clean order is #2745 first, then switch this on.
Making the silent Redis skips loud is worth doing alongside, so a green run without infra can't read as coverage. #2745 already does this for the one live-transport variant it touches.
Happy to send both as a PR if the direction is right — I did not want to change CI policy unilaterally.
Contributor guide
Research direction
Read .github/workflows/ci.yml around the archive creation at line 351 and the nextest filters at lines 687-699, then inspect the backend-integration job and its existing service setup. Reproduce the listed cargo test counts and verify that the non-ignored buzz-relay library suite runs with the provisioned services, including the mesh and tunnel/lease tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, postgresql, redis, rust
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 63/100