Relay: reconnects fail ~22% of the time on hosted communities (500 in auth path, 404 on WS upgrade)
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
Reporting measurements from a hosted community (`*.communities.buzz.xyz`), gathered from `buzz-acp` harness logs on one macOS machine running Buzz Desktop 0.5.5 across ~6 days (2026-08-01 → 08-06). Happy to supply raw logs.
## 1. Long-lived WebSockets are recycled about every 15–17 minutes
420 `relay connection lost — reconnecting…` events across the agent sessions on this machine. Histogram of the gap between consecutive drops per session:
```
gaps in the 14–21 min window: 186 of 420 (median 17.1 min)
15m30s : ################## (18)
16m00s : ######################################### (41)
16m30s : ########################### (27)
17m00s : ########################## (26)
17m30s : ################### (19)
18m00s : #############################(29)
18m30s : ####### (7)
19m00s : ########## (10)
```
Note the hard floor at 15m30s — nothing at 13–15 min at all — with a tail upward. That is the shape of a fixed max connection age (~900s) observed on the client's next heartbeat, not random instability. Drops are spread evenly across sessions (87 / 88 / 90) and are mostly *not* simultaneous — only 3 seconds in the whole window had 3+ sessions drop together — so this looks per-connection rather than a server restart.
If that recycling is intentional (proxy/LB max age), this part is just FYI — the client handles it and most reconnects land in under a second. The next part is the actual problem.
## 2. About one reconnect in five fails, and the relay is the one erroring
```
528 reconnect attempts
114 failures (21.6%)
12 occasions where all 5 attempts were exhausted
```
Failure reasons:
| count | error |
|---|---|
| 82 | `Connection closed` |
| 11 | `Auth failed: error: internal error checking restriction state` |
| 9 | `WebSocket protocol error: Connection reset without closing handshake` |
| 8 | `WebSocket error: HTTP error: 404 Not Found` |
| 4 | `IO error: Connection reset by peer (os error 54)` |
Two of these are generated by the relay itself and look like genuine server-side bugs:
- **`internal error checking restriction state`** (11×) — a 500 out of the auth path. The client is authenticating with the same key that worked moments earlier.
- **`HTTP error: 404 Not Found`** (8×) — a 404 on the WebSocket upgrade for a relay URL that is otherwise valid and serving.
A representative cascade where every attempt failed against a different error:
```
17:10:31Z reconnect attempt 1 failed: Connection closed
17:10:41Z reconnect attempt 2 failed: Auth failed: error: internal error checking restriction state
17:10:49Z reconnect attempt 3 failed: WebSocket protocol error: Connection reset without closing handshake
17:10:56Z reconnect attempt 4 failed: WebSocket error: HTTP error: 404 Not Found
17:11:12Z reconnect attempt 5 failed: Connection closed
```
The mix of 500s, 404s and resets inside 40 seconds reads like requests landing on an instance that is not ready to serve — but the auth-path 500 in particular seems worth a look on its own, since it is the relay reporting an internal failure rather than a rejection.
## Impact
`buzz-acp` gives up after 5 attempts, so those 12 exhaustions are windows where an agent session is off the relay until something re-engages it. Combined with #1, every long-lived client is re-rolling this dice roughly 4×/hour.
## Related
#5029 addresses the *client-side presentation* of these blips (the sidebar card currently says "Can't reach the relay / Click to connect" while the backoff loop is mid-retry). It does not address the failures above.
Contributor guide
Research direction
Start with the `buzz-acp` harness logs and trace the relay authentication path and WebSocket upgrade handling described in the report. Compare the 500 restriction-state errors, 404 upgrades, resets, and connection recycling against the behavior noted in #5029. Done means the server-side causes are identified and reconnect exhaustion is no longer reproducible in the reported scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100