block / block/buzz

Relay: reconnects fail ~22% of the time on hosted communities (500 in auth path, 404 on WS upgrade)

Open
#5,030 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

Reporting measurements from a hosted community (`*.communities.buzz.xyz`), gathered from `buzz-acp` harness logs on one macOS machine running Buzz Desktop 0.5.5 across ~6 days (2026-08-01 → 08-06). Happy to supply raw logs.

## 1. Long-lived WebSockets are recycled about every 15–17 minutes

420 `relay connection lost — reconnecting…` events across the agent sessions on this machine. Histogram of the gap between consecutive drops per session:

```
gaps in the 14–21 min window: 186 of 420 (median 17.1 min)

15m30s : ################## (18)
16m00s : ######################################### (41)
16m30s : ########################### (27)
17m00s : ########################## (26)
17m30s : ################### (19)
18m00s : #############################(29)
18m30s : ####### (7)
19m00s : ########## (10)
```

Note the hard floor at 15m30s — nothing at 13–15 min at all — with a tail upward. That is the shape of a fixed max connection age (~900s) observed on the client's next heartbeat, not random instability. Drops are spread evenly across sessions (87 / 88 / 90) and are mostly *not* simultaneous — only 3 seconds in the whole window had 3+ sessions drop together — so this looks per-connection rather than a server restart.

If that recycling is intentional (proxy/LB max age), this part is just FYI — the client handles it and most reconnects land in under a second. The next part is the actual problem.

## 2. About one reconnect in five fails, and the relay is the one erroring

```
528 reconnect attempts
114 failures (21.6%)
12 occasions where all 5 attempts were exhausted
```

Failure reasons:

| count | error |
|---|---|
| 82 | `Connection closed` |
| 11 | `Auth failed: error: internal error checking restriction state` |
| 9 | `WebSocket protocol error: Connection reset without closing handshake` |
| 8 | `WebSocket error: HTTP error: 404 Not Found` |
| 4 | `IO error: Connection reset by peer (os error 54)` |

Two of these are generated by the relay itself and look like genuine server-side bugs:

- **`internal error checking restriction state`** (11×) — a 500 out of the auth path. The client is authenticating with the same key that worked moments earlier.
- **`HTTP error: 404 Not Found`** (8×) — a 404 on the WebSocket upgrade for a relay URL that is otherwise valid and serving.

A representative cascade where every attempt failed against a different error:

```
17:10:31Z reconnect attempt 1 failed: Connection closed
17:10:41Z reconnect attempt 2 failed: Auth failed: error: internal error checking restriction state
17:10:49Z reconnect attempt 3 failed: WebSocket protocol error: Connection reset without closing handshake
17:10:56Z reconnect attempt 4 failed: WebSocket error: HTTP error: 404 Not Found
17:11:12Z reconnect attempt 5 failed: Connection closed
```

The mix of 500s, 404s and resets inside 40 seconds reads like requests landing on an instance that is not ready to serve — but the auth-path 500 in particular seems worth a look on its own, since it is the relay reporting an internal failure rather than a rejection.

## Impact

`buzz-acp` gives up after 5 attempts, so those 12 exhaustions are windows where an agent session is off the relay until something re-engages it. Combined with #1, every long-lived client is re-rolling this dice roughly 4×/hour.

## Related

#5029 addresses the *client-side presentation* of these blips (the sidebar card currently says "Can't reach the relay / Click to connect" while the backoff loop is mid-retry). It does not address the failures above.

Contributor guide

Open the contributing guide

Research direction

Start with the `buzz-acp` harness logs and trace the relay authentication path and WebSocket upgrade handling described in the report. Compare the 500 restriction-state errors, 404 upgrades, resets, and connection recycling against the behavior noted in #5029. Done means the server-side causes are identified and reconnect exhaustion is no longer reproducible in the reported scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.