relay: production panic in reaction ingest — channel_id.expect("reaction path has channel") fired (ingest.rs:2258)
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
**What happened**
At 2026-07-22 07:00:34 UTC, a production relay pod (bb-public cell) logged:
```
thread 'tokio-rt-worker' (7) panicked at crates/buzz-relay/src/handlers/ingest.rs:2258:51:
reaction path has channel
```
Immediately preceded by the `reaction path has channel` expect message and `note: run with RUST_BACKTRACE=1...`. The process did not restart (tokio caught the task panic), so blast radius was one client's reaction request — but the reaction row had already been inserted by `insert_reaction_event_with_thread_metadata` before the panic point, so the client likely got a dropped connection with the write already committed (re-send would then hit the Duplicate path).
**The bug**
`handle_reaction` (ingest.rs, current main `7e34bee62`) emits trace actions with:
```rust
channel: channel_label(channel_id.expect("reaction path has channel")),
```
at lines 2258 and 2264, under a comment asserting "channel_id is always Some here, so WriteInsertGlobal does not apply." Production disproved the invariant: at least one kind:7 event reaches this point with `channel_id == None` (reaction to a target addressed without an `h` tag / non-channel context — exact trigger event unknown; it happened once in ~17h of launch traffic).
**Suggested fix**
Don't panic in the trace-emit path. Either map `None` to a sentinel label (e.g. `channel_label_opt` / `"none"`), or emit `WriteInsertGlobal` when `channel_id` is `None`, matching whatever the trace spec says for channel-less writes. Trace emission should never be able to kill the task after the DB write has committed.
**Evidence**
- Datadog: cluster `production-bke-bb-public-infra-cell-01-us-west-2`, ns `buzz`, host `i-00e17d2517638dafc`, 2026-07-22T07:00:34Z ("panicked at crates/buzz-relay/src/handlers/ingest.rs:2258:51" + "reaction path has channel" adjacent line)
- Single occurrence so far; relay restart count unaffected.
Contributor guide
Research direction
Start in crates/buzz-relay/src/handlers/ingest.rs around handle_reaction and lines 2258 and 2264, then inspect insert_reaction_event_with_thread_metadata and the trace specification for channel-less writes. Reproduce or cover a kind:7 reaction with channel_id == None, and verify trace emission does not panic after the database write while following the specified handling for the missing channel.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100