Hosted relay: synchronised WebSocket RSTs across sockets of differing ages (upstream connection recycling)
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
### Summary
On the hosted relay (`.communities.buzz.xyz`), WebSocket connections are reset by the server in **synchronised bursts that hit sockets of wildly different ages at the same instant**. This looks like upstream connection recycling (Cloudflare edge and/or the Envoy fleet in front of the relay) rather than anything client-side.
Related to #4749. The user-visible damage is amplified by #4752 (mentions published during the reconnect gap are silently lost).
### Environment
- Buzz Desktop **0.5.5** (`desktop-v0.5.5`)
- macOS 26 (Darwin 27), Apple Silicon
- 8 managed agents, each holding its own relay WebSocket, plus the desktop (~9–12 sockets from one IP)
- Relay: hosted `*.communities.buzz.xyz`
### The evidence
Every drop is `WebSocket protocol error: Connection reset without closing handshake` — the peer kills the TCP connection without sending a WS Close frame.
**Sockets of very different ages die in the same second, while other sockets on the same host survive:**
| Time (UTC) | Sockets reset together | Ages at death |
|---|---|---|
| `14:43:25Z` | Fable 5 + Honey + Kimi | **7m**, **98m**, **158m** |
| `13:05:32Z` | Cursor + Honey | 60m35s, 60m35s |
| `18:57:11Z` | Grok + Honey | 10m41s, 17m51s (30 ms apart) |
| `19:06:45Z` | Fizz + Fable 5 | (61 ms apart) |
A client-side, Wi-Fi, or NAT-idle cause cannot synchronise the death of a 7-minute-old and a 158-minute-old socket to the same second while leaving 6 other sockets on the same host alive. That pattern requires a shared upstream component holding a *subset* of the connections — i.e. one proxy instance or relay replica draining.
Response headers confirm two middleboxes in the path:
```
server: cloudflare
cf-ray: …-MAA
x-envoy-upstream-service-time: …
```
An evening cluster showed one cohort of 8 sockets (all born within 6 seconds) being picked off in singles and pairs over 32 minutes at ages 165–197m — consistent with rolling drains across instances, not a fixed connection TTL.
### Ruled out client-side (with data)
| Hypothesis | Result |
|---|---|
| Relay rate limiting | Refuted. All `rate-limited: quota exceeded` notices occurred in one 64s burst after a mass agent restart; **zero** within hours of any reset. |
| `--relay-observer` traffic | Refuted. The heaviest-traffic window of the day had **zero** resets; a near-idle hour had two. No correlation in either direction. |
| Wi-Fi / link instability | Refuted. `log show` (airportd/configd) shows zero link/roam/DHCP/disassoc events across a 40-minute window in which 8 sockets were reset. |
| Heavy agent activity / large messages | Refuted — inversely correlated if anything. |
| Client ping/pong bug | Refuted. The pong watchdog (`no pong received within 10s`) fired **zero** times on the affected day; these were active resets, not late-detected half-open sockets. |
| Host sleep | Separated out. Sleep-correlated drops are a distinct population with different signatures (`os error 60`, `os error 49`). |
### Impact
Reconnects themselves are reliable (114 successes, 0 retry exhaustions in one day). The cost isn't the reconnect — it's #4752: because resubscribe filters from `now` rather than last-seen, **any mention published during the ~1.4s gap is silently dropped**, with no error surfaced to the user. On a busy multi-agent workspace this is a real, invisible work-loss path.
### Suggested direction
1. Server-side: if connection recycling is intentional (drains, config reloads), send a WebSocket Close frame with a status code so clients can distinguish a planned drain from a network fault.
2. Client-side: make resubscribe resume from **last-seen event** rather than `now` (#4752). That alone would make these blips harmless regardless of what upstream does.
3. Possibly related: #4667 (zombie in-flight turn after reconnect + rate-limit), #3974 (no idle-read watchdog for half-open sockets).
Happy to share the exact relay hostname, `cf-ray` values and full timestamped logs privately if that helps correlate against server-side records.
Contributor guide
Research direction
Start with #4752, then trace the hosted relay WebSocket reconnect and resubscribe path; the issue provides no repository file or test names. Compare reset timestamps and close/error handling against the listed hypotheses. Done requires an agreed server-side close behavior or recovery change that prevents mentions during reconnect gaps from being lost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100