block / block/buzz

buzz-relay lib tests are archived but not run in CI — 771 of 804 never execute, including the mesh and tunnel/lease suites

Open
#3,461 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

`buzz-relay`'s library tests are compiled in CI but almost none are executed. Of 804 tests in `cargo test -p buzz-relay --lib`, CI runs **12** — the `#[ignore]`d subset of `api::invites::tests`. All **771 non-ignored** tests, including the entire mesh, tunnel/lease, subscription, media-auth, and pubsub-fanout suites, execute nowhere in any workflow.

This looks unintentional rather than a deliberate policy: the backend-integration job already provisions Postgres, Redis, and MinIO, and the nextest archive is explicitly built with `-p buzz-relay --lib`. The filter that selects tests from that archive was written for the invite suite and appears never to have been broadened.

## Evidence

Every test invocation across `.github/workflows/`:

| Location | Runs |
|---|---|
| `ci.yml:129` | `just test-unit` → `buzz-core`, `buzz-auth`, `buzz-db --lib`, `buzz-conformance`, `buzz-push-gateway` (justfile:275). No `buzz-relay`. |
| `ci.yml:687` | archive, `-E '(package(buzz-db) and test(/relay_invite::tests/)) or (package(buzz-relay) and test(/api::invites::tests/))' --run-ignored ignored-only` |
| `ci.yml:699` | archive, `-E 'binary(e2e_event_reminder)' --run-ignored ignored-only` |
| `ci.yml:740-742` | `buzz-test-client` e2e (persona, nostr_interop, relay invite, nip43) |
| `ci.yml:970`, `ci.yml:1005` | `buzz-dev-mcp`, desktop Tauri (Windows) |

The archive is created at `ci.yml:351` with `-p buzz-db -p buzz-relay -p buzz-test-client --lib`, so the binaries exist; only those two `-E` filters ever select from it, and both pass `--run-ignored ignored-only`.

Reproduce the counts:

```bash
cargo test -p buzz-relay --lib -- --list | grep -c ': test$' # 804
cargo test -p buzz-relay --lib -- --list --ignored | grep -c ': test$' # 33
cargo test -p buzz-relay --lib -- --list --ignored | grep -c 'api::invites' # 12 ← what CI runs
```

## Why it matters

This is the reason #2458 survived in `main`. `api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo` fails roughly half the time on a machine with Redis — I measured 5/10 failures on `origin/main` @ 22be8bb3, and @tlongwell-block and two others reproduced it independently this week. A test with that failure rate cannot survive a CI that runs it. It survived because CI never has.

The same blind spot covers `tunnel::directory` (lease acquisition, fencing, TTL takeover) and `tunnel::reliable` (ownership routing, renewer loss signalling) — the coordination primitives underneath the mesh.

There is a second, related trap that @tlongwell-block flagged on #2458: 11 of these tests call `redis_directory_if_available()` and **return early, silently, and green** when Redis is unreachable (`tunnel/directory.rs` ×4, `tunnel/reliable.rs` ×4, `api/mesh_demo.rs` ×2, `handlers/event.rs` ×1). So even running them locally without infra proves nothing, and the output is indistinguishable from real coverage.

## Cost to fix

Approximately zero. The full non-ignored suite against local Postgres + Redis:

```
test result: ok. 771 passed; 0 failed; 33 ignored; 0 measured; 0 filtered out; finished in 2.22s
```

2.22 seconds, all green, in a job that already has the services running.

## Suggested fix

Add a step to `backend-integration` (which already has Postgres/Redis/MinIO up and the archive downloaded) running the `buzz-relay` lib suite from the existing archive.

Ordering matters: enabling this today makes CI flake about half the time on the #2458 test. #2745 fixes that flake — I isolated the mechanism to cancel-safety in `run_demo_echo` with a 2×2 matrix at N=10 per cell ([detail](https://github.com/block/buzz/issues/2458#issuecomment-5112532129)) — so the clean order is #2745 first, then switch this on.

Making the silent Redis skips loud is worth doing alongside, so a green run without infra can't read as coverage. #2745 already does this for the one live-transport variant it touches.

Happy to send both as a PR if the direction is right — I did not want to change CI policy unilaterally.

Contributor guide

Open the contributing guide

Research direction

Read .github/workflows/ci.yml around the archive creation at line 351 and the nextest filters at lines 687-699, then inspect the backend-integration job and its existing service setup. Reproduce the listed cargo test counts and verify that the non-ignored buzz-relay library suite runs with the provisioned services, including the mesh and tunnel/lease tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, postgresql, redis, rust
Domain
ci-cd, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
63/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.