relay: live fan-out can deliver EVENT for a subscription ID it no longer owns
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Problem
The relay can deliver a live `EVENT` frame for a subscription ID that no longer belongs to the request that produced it. A WebSocket client can therefore receive an event it did not ask for, attributed to a subscription it does own, or receive an `EVENT` after the relay has already told it that subscription is `CLOSED`.
## Mechanism
Live fan-out selects its recipients, then awaits access resolution, then queues frames — without revalidating ownership at the moment of insertion.
1. `fan_out_event_to_local_subscribers` snapshots matching `(conn_id, sub_id)` pairs from the registry (`crates/buzz-relay/src/handlers/event.rs:246`; the same shape appears at `:306` and `:431`).
2. `filter_fanout_by_access` awaits channel-visibility and membership lookups (`crates/buzz-relay/src/handlers/event.rs:189-212`).
3. During that await, the subscription's lifecycle can cut over: either `CLOSE x` removes it and queues the `CLOSED` acknowledgement (`crates/buzz-relay/src/handlers/close.rs:24-50`), or a newer `REQ` reusing ID `x` commits and replaces the registry entry (`crates/buzz-relay/src/handlers/req.rs:560-611`).
4. Fan-out resumes and queues the frame from its stale snapshot via `send_fanout_frames` (`crates/buzz-relay/src/handlers/event.rs:76-92`), which calls `ConnectionManager::send_to_text_bytes`. That validates only that the connection exists and its buffer is not full (`crates/buzz-relay/src/state.rs:586-605`) — not subscription generation.
Registry ownership is checked when the recipient list is built, not when the frame is inserted into the outbound queue. The await turns the recipient list into a stale capability.
Two observable outcomes:
- **After replacement:** the client sees an `EVENT` matching the *old* filter, labelled with the ID the *new* subscription now owns. It cannot distinguish the two.
- **After CLOSE:** the client sees `EVENT x` arrive after `CLOSED x`, violating terminal-frame ordering.
## Second instance: access-revocation eviction
`evict_conn_channel_subscriptions` removes registry entries, later awaits the connection map, then releases topics and queues a raw `CLOSED` (`crates/buzz-relay/src/handlers/side_effects.rs:100-137`). If a same-ID `REQ` commits during that window, eviction resumes and removes the newer subscription from the per-connection map and queues a `CLOSED` naming its ID, while the newer entry remains in the registry. The topic refcount can also be decremented for the evicted subscription after the newer one's retain.
## Scope and provenance
This is pre-existing behavior on `main`, not a regression. It was found while reviewing #2382, which fences the `REQ` lease family — pre-registration rejection, registration commit, historical replay, `CLOSE` acknowledgement, same-ID replacement, disconnect drain — under a per-connection lifecycle lock. At #2382's head `e8e0fcc34afe637530efd1e260bd75db5235d174`, `handlers/event.rs`, `handlers/side_effects.rs`, and `subscription.rs` are byte-identical to `main` apart from one added line in an `event.rs` test fixture. #2382 was deliberately scoped to the request lease and does not address live fan-out; this issue tracks that remaining gap.
## Suggested direction
Generation-aware live delivery: carry a subscription generation or lease handle on registry recipients and revalidate it at queue-insertion time, sharing the serialization boundary that `CLOSE` and replacement commit already use. Revocation eviction should participate in the same per-ID transaction or carry the same generation.
## Acceptance
Deterministic regressions that pause live fan-out *after* recipient selection and prove no stale frame is queued, for all three orderings: fan-out vs same-ID replacement, fan-out vs `CLOSE`, and revocation eviction vs same-ID replacement.
## Credit
Mechanism, orderings, and citations independently identified and corroborated by two reviewers during the #2382 concurrency review.
Contributor guide
Assessment
This issue has not been assessed yet.