block / block/buzz

CI never runs the buzz-db replica-routing Postgres fixtures — enforce them before enabling BUZZ_REPLICA_READ_MAX_AGE_MS

Open
#3,622 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

None of the Postgres-backed replica-routing fixtures added in #3268 — nor most of the 34 `#[ignore]`d tests in `crates/buzz-db/src/lib.rs` — are ever executed by CI. They were verified exhaustively by hand at `dd26caa9f` (see the PR body), but a future regression on the routed-read path can merge green.

Found by Dawn during her #3268 sign-off amendment; independently confirmed by Max and Eva at the same SHA.

## Evidence (at `dd26caa9f`)

- `Unit Tests` runs `just test-unit` → `cargo nextest run -p buzz-db --lib` (`Justfile:279-285`). Ignored tests are excluded by design, and the Justfile comment says so.
- The only `--run-ignored ignored-only` nextest invocations in `.github/workflows/` are `ci.yml:690` and `:702`, both filtered to `(package(buzz-db) and test(/relay_invite::tests/)) or (package(buzz-relay) and test(/api::invites::tests/))`.
- Therefore, of the 34 `#[ignore]`d buzz-db tests, CI runs only the `relay_invite::tests` subset. The one-budget acquire regression (`routed_fallback_spends_one_acquire_budget_when_aurora_cache_is_cold`), the seven-seam community-isolation fixture, and all fence/floor-guard/fallback fixtures have **zero automated execution**.

## Why this matters

- The one-budget fixture exists precisely because a "fix is present" claim survived review without a measurement (#3268 thread, Dawn's Pass 11 postmortem). An `#[ignore]`d test no job selects is a comment, not a gate.
- Max's rollout condition: **do not set `BUZZ_REPLICA_READ_MAX_AGE_MS` on bb-block until this is fixed.** Merge itself is safe (new seams are dark by default), but enabling the new seams in production without CI enforcement of their regressions is not.

## Constraints on the fix

1. **Ordering dependency on #3619**: the moment any filter widening selects `test_usage_metrics_lock_has_single_owner_and_releases_on_drop`, the shared-DB advisory-lock collision fires in CI. #3619 (scratch-DB isolation for that test) must land first, or the selection must explicitly exclude it.
2. **A naive regex is insufficient.** Dawn's first-draft filter `test(/tests::(routed|replica|reader|fence|floor_guard)/)` misses at least: `query_events_routed_*`, `count_events_routed_*`, `lazy_reader_pool_*`, `head_fetch_routes_by_configured_budget`, `channel_window_routes_*`, `thread_replies_cursor_pages_*`, `read_session_degrades_*`. A partially-covering filter recreates this bug one level down — green CI, unenforced invariants.
3. Several fixtures assume scratch DBs and serial execution; the selection should run serially against the existing `backend-integration` Postgres archive (that job already has the infra).

## Suggested shape (per Max)

An **explicit, auditable selection** rather than a clever regex: either move the replica-routing fixtures into a dedicated test module (e.g. `tests::replica_routing::`) and filter on that module, or maintain an explicit nextest filter list in `ci.yml` with a comment stating it must cover every `#[ignore]`d routing fixture. Initially cover the exact routing matrix verified by hand in #3268. Keep `test_usage_metrics_lock...` excluded until #3619 lands.

## Acceptance

- The one-budget fixture and the seven-seam isolation fixture demonstrably run in CI (visible in the job log at least once).
- A revert of only the #3268 production fix fails CI (the 320ms defang check, now automated).
- No selection of the usage-metrics lock test until #3619 is closed.

Refs: #3268, #3619.

Contributor guide

Open the contributing guide

Research direction

Start by reading .github/workflows/ci.yml:690-702, Justfile:279-285, and the ignored tests in crates/buzz-db/src/lib.rs. Run the existing backend-integration Postgres job or inspect its nextest selection, then verify that the one-budget and seven-seam fixtures appear in its log while test_usage_metrics_lock_has_single_owner_and_releases_on_drop remains excluded. Done means the listed routing matrix is explicitly covered and the acceptance checks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, postgresql, rust
Domain
ci-cd, databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.