block / block/buzz

relay: production panic in reaction ingest — channel_id.expect("reaction path has channel") fired (ingest.rs:2258)

Open
#2,348 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

**What happened**
At 2026-07-22 07:00:34 UTC, a production relay pod (bb-public cell) logged:

```
thread 'tokio-rt-worker' (7) panicked at crates/buzz-relay/src/handlers/ingest.rs:2258:51:
reaction path has channel
```

Immediately preceded by the `reaction path has channel` expect message and `note: run with RUST_BACKTRACE=1...`. The process did not restart (tokio caught the task panic), so blast radius was one client's reaction request — but the reaction row had already been inserted by `insert_reaction_event_with_thread_metadata` before the panic point, so the client likely got a dropped connection with the write already committed (re-send would then hit the Duplicate path).

**The bug**
`handle_reaction` (ingest.rs, current main `7e34bee62`) emits trace actions with:

```rust
channel: channel_label(channel_id.expect("reaction path has channel")),
```

at lines 2258 and 2264, under a comment asserting "channel_id is always Some here, so WriteInsertGlobal does not apply." Production disproved the invariant: at least one kind:7 event reaches this point with `channel_id == None` (reaction to a target addressed without an `h` tag / non-channel context — exact trigger event unknown; it happened once in ~17h of launch traffic).

**Suggested fix**
Don't panic in the trace-emit path. Either map `None` to a sentinel label (e.g. `channel_label_opt` / `"none"`), or emit `WriteInsertGlobal` when `channel_id` is `None`, matching whatever the trace spec says for channel-less writes. Trace emission should never be able to kill the task after the DB write has committed.

**Evidence**
- Datadog: cluster `production-bke-bb-public-infra-cell-01-us-west-2`, ns `buzz`, host `i-00e17d2517638dafc`, 2026-07-22T07:00:34Z ("panicked at crates/buzz-relay/src/handlers/ingest.rs:2258:51" + "reaction path has channel" adjacent line)
- Single occurrence so far; relay restart count unaffected.

Contributor guide

Open the contributing guide

Research direction

Start in crates/buzz-relay/src/handlers/ingest.rs around handle_reaction and lines 2258 and 2264, then inspect insert_reaction_event_with_thread_metadata and the trace specification for channel-less writes. Reproduce or cover a kind:7 reaction with channel_id == None, and verify trace emission does not panic after the database write while following the specified handling for the missing channel.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.