Intermittent 403 on POST /events (HTTP bridge) causes buzz-agent deadlock
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
When running `buzz-acp` agents that send messages via the HTTP bridge (`POST /events`), the relay intermittently returns **403 Forbidden** for events that should be accepted. The 403 occurs mid-turn for an already-authenticated pubkey, with successful `/events` calls before and after the rejected one within the same second. When `BUZZ_REQUIRE_AUTH_TOKEN=true`, this 403 causes `buzz-agent` to deadlock (the ACP state machine waits for a tool result that never arrives), leaving the agent slot permanently occupied until manual restart.
## Environment
- **Relay image**: `ghcr.io/block/buzz:main` (built 2026-07-25)
- **Config**: `BUZZ_REQUIRE_RELAY_MEMBERSHIP=true`, `BUZZ_ALLOW_NIP_OA_AUTH=true`
- **Agents**: 3 × `buzz-acp` systemd units, each with `agents=1`, `subscribe=mentions`, `context_limit=100`
- **Auth**: WebSocket via NIP-42 (successful); HTTP bridge via NIP-98
## Evidence
### Pattern: 403 sandwiched between successful calls
The same authenticated pubkey gets 200 → 403 → 200 within the same second:
```json
{"timestamp":"…T03:48:00.092Z","route":"/events","pubkey":"f852c3e7…","status":200,"accepted":true}
{"timestamp":"…T03:48:04.265Z","route":"/events","pubkey":"f852c3e7…","status":403,"accepted":false}
{"timestamp":"…T03:48:04.273Z","route":"/query","pubkey":"f852c3e7…","status":200,"result_count":2}
{"timestamp":"…T03:48:04.295Z","route":"/events","pubkey":"f852c3e7…","status":200,"accepted":true}
```
The 403 is **not** an auth failure — the surrounding calls from the same pubkey succeed. It appears to be a specific event being rejected, but the relay logs no `reason` field explaining why.
### Frequency
Over ~2 hours of active use (with `BUZZ_REQUIRE_AUTH_TOKEN=true`):
- 63 × `status:200` on `/events`
- 13 × `status:403` on `/events` (**17% rejection rate**)
The most active agent (most frequent HTTP bridge calls) received 10 of the 13 rejections.
### Impact: buzz-agent deadlock
When a tool call (e.g., `buzz messages send`) receives a 403, `buzz-agent` enters a state where:
- The agent subprocess blocks on `read(stdin)` waiting for the next ACP prompt block
- The ACP harness blocks on a futex waiting for the agent's response
- `idle_timeout` (120s) does **not** fire — the watchdog appears to only cover `read_until_response_with_idle_timeout`, not the gap between a failed tool result and the next LLM call
The only recovery is `systemctl restart`.
## What we tried
| Change | Effect on 403 | Effect on deadlock |
|---|---|---|
| `BUZZ_REQUIRE_AUTH_TOKEN=true → false` | 403 still occurs (~14% rate) | **Deadlock no longer occurs** — agent retries and recovers |
| `CONTEXT_MESSAGE_LIMIT` 100 → 50 | No effect | No effect |
| Model swap (LongCat-2.0 → qwen3.6-flash) | No effect | No effect |
Disabling `BUZZ_REQUIRE_AUTH_TOKEN` prevents the deadlock but doesnt eliminate the 403 itself.
## Asks
1. **Root cause**: What condition triggers a 403 on `POST /events` for an authenticated, channel-member pubkey? The relay logs `"status":403` with no `reason` — adding a rejection reason would help diagnosis.
2. **Is this a membership cache race?** The same pubkey succeeds before and after the 403 within 1 second, suggesting a transient membership or auth-state lookup issue rather than a genuine authorization failure.
3. **Should the HTTP bridge retry on 403?** If the rejection is transient, an automatic retry (similar to the WebSocket path) would prevent downstream deadlocks.
## Reproduction
1. Self-host relay with `BUZZ_REQUIRE_RELAY_MEMBERSHIP=true`, `BUZZ_REQUIRE_AUTH_TOKEN=true`
2. Run 3 `buzz-acp` agents subscribed to a channel via NIP-42 WS
3. `@mention` all three simultaneously
4. Observe relay logs: intermittent 403 on `POST /events` from agent pubkeys
5. With `AUTH_TOKEN=true`: affected agent deadlocks permanently
Contributor guide
Research direction
Start by reproducing the intermittent POST /events rejection with BUZZ_REQUIRE_RELAY_MEMBERSHIP=true and BUZZ_REQUIRE_AUTH_TOKEN=true, then trace the relay's /events authorization path and the HTTP bridge handling of a 403. Compare the rejected event with surrounding successful calls and inspect the ACP wait path. Done means the rejection condition is identified, the relay explains 403 responses, and the agent no longer deadlocks after a failed tool call.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- authentication, backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100