block / block/buzz

Intermittent 403 on POST /events (HTTP bridge) causes buzz-agent deadlock

Open
#2,936 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

When running `buzz-acp` agents that send messages via the HTTP bridge (`POST /events`), the relay intermittently returns **403 Forbidden** for events that should be accepted. The 403 occurs mid-turn for an already-authenticated pubkey, with successful `/events` calls before and after the rejected one within the same second. When `BUZZ_REQUIRE_AUTH_TOKEN=true`, this 403 causes `buzz-agent` to deadlock (the ACP state machine waits for a tool result that never arrives), leaving the agent slot permanently occupied until manual restart.

## Environment

- **Relay image**: `ghcr.io/block/buzz:main` (built 2026-07-25)
- **Config**: `BUZZ_REQUIRE_RELAY_MEMBERSHIP=true`, `BUZZ_ALLOW_NIP_OA_AUTH=true`
- **Agents**: 3 × `buzz-acp` systemd units, each with `agents=1`, `subscribe=mentions`, `context_limit=100`
- **Auth**: WebSocket via NIP-42 (successful); HTTP bridge via NIP-98

## Evidence

### Pattern: 403 sandwiched between successful calls

The same authenticated pubkey gets 200 → 403 → 200 within the same second:

```json
{"timestamp":"…T03:48:00.092Z","route":"/events","pubkey":"f852c3e7…","status":200,"accepted":true}
{"timestamp":"…T03:48:04.265Z","route":"/events","pubkey":"f852c3e7…","status":403,"accepted":false}
{"timestamp":"…T03:48:04.273Z","route":"/query","pubkey":"f852c3e7…","status":200,"result_count":2}
{"timestamp":"…T03:48:04.295Z","route":"/events","pubkey":"f852c3e7…","status":200,"accepted":true}
```

The 403 is **not** an auth failure — the surrounding calls from the same pubkey succeed. It appears to be a specific event being rejected, but the relay logs no `reason` field explaining why.

### Frequency

Over ~2 hours of active use (with `BUZZ_REQUIRE_AUTH_TOKEN=true`):
- 63 × `status:200` on `/events`
- 13 × `status:403` on `/events` (**17% rejection rate**)

The most active agent (most frequent HTTP bridge calls) received 10 of the 13 rejections.

### Impact: buzz-agent deadlock

When a tool call (e.g., `buzz messages send`) receives a 403, `buzz-agent` enters a state where:
- The agent subprocess blocks on `read(stdin)` waiting for the next ACP prompt block
- The ACP harness blocks on a futex waiting for the agent's response
- `idle_timeout` (120s) does **not** fire — the watchdog appears to only cover `read_until_response_with_idle_timeout`, not the gap between a failed tool result and the next LLM call

The only recovery is `systemctl restart`.

## What we tried

| Change | Effect on 403 | Effect on deadlock |
|---|---|---|
| `BUZZ_REQUIRE_AUTH_TOKEN=true → false` | 403 still occurs (~14% rate) | **Deadlock no longer occurs** — agent retries and recovers |
| `CONTEXT_MESSAGE_LIMIT` 100 → 50 | No effect | No effect |
| Model swap (LongCat-2.0 → qwen3.6-flash) | No effect | No effect |

Disabling `BUZZ_REQUIRE_AUTH_TOKEN` prevents the deadlock but doesnt eliminate the 403 itself.

## Asks

1. **Root cause**: What condition triggers a 403 on `POST /events` for an authenticated, channel-member pubkey? The relay logs `"status":403` with no `reason` — adding a rejection reason would help diagnosis.
2. **Is this a membership cache race?** The same pubkey succeeds before and after the 403 within 1 second, suggesting a transient membership or auth-state lookup issue rather than a genuine authorization failure.
3. **Should the HTTP bridge retry on 403?** If the rejection is transient, an automatic retry (similar to the WebSocket path) would prevent downstream deadlocks.

## Reproduction

1. Self-host relay with `BUZZ_REQUIRE_RELAY_MEMBERSHIP=true`, `BUZZ_REQUIRE_AUTH_TOKEN=true`
2. Run 3 `buzz-acp` agents subscribed to a channel via NIP-42 WS
3. `@mention` all three simultaneously
4. Observe relay logs: intermittent 403 on `POST /events` from agent pubkeys
5. With `AUTH_TOKEN=true`: affected agent deadlocks permanently

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the intermittent POST /events rejection with BUZZ_REQUIRE_RELAY_MEMBERSHIP=true and BUZZ_REQUIRE_AUTH_TOKEN=true, then trace the relay's /events authorization path and the HTTP bridge handling of a 403. Compare the rejected event with surrounding successful calls and inspect the ACP wait path. Done means the rejection condition is identified, the relay explains 403 responses, and the agent no longer deadlocks after a failed tool call.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
authentication, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.