block / block/buzz

Hosted relay: synchronised WebSocket RSTs across sockets of differing ages (upstream connection recycling)

Open
#4,944 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

### Summary

On the hosted relay (`.communities.buzz.xyz`), WebSocket connections are reset by the server in **synchronised bursts that hit sockets of wildly different ages at the same instant**. This looks like upstream connection recycling (Cloudflare edge and/or the Envoy fleet in front of the relay) rather than anything client-side.

Related to #4749. The user-visible damage is amplified by #4752 (mentions published during the reconnect gap are silently lost).

### Environment

- Buzz Desktop **0.5.5** (`desktop-v0.5.5`)
- macOS 26 (Darwin 27), Apple Silicon
- 8 managed agents, each holding its own relay WebSocket, plus the desktop (~9–12 sockets from one IP)
- Relay: hosted `*.communities.buzz.xyz`

### The evidence

Every drop is `WebSocket protocol error: Connection reset without closing handshake` — the peer kills the TCP connection without sending a WS Close frame.

**Sockets of very different ages die in the same second, while other sockets on the same host survive:**

| Time (UTC) | Sockets reset together | Ages at death |
|---|---|---|
| `14:43:25Z` | Fable 5 + Honey + Kimi | **7m**, **98m**, **158m** |
| `13:05:32Z` | Cursor + Honey | 60m35s, 60m35s |
| `18:57:11Z` | Grok + Honey | 10m41s, 17m51s (30 ms apart) |
| `19:06:45Z` | Fizz + Fable 5 | (61 ms apart) |

A client-side, Wi-Fi, or NAT-idle cause cannot synchronise the death of a 7-minute-old and a 158-minute-old socket to the same second while leaving 6 other sockets on the same host alive. That pattern requires a shared upstream component holding a *subset* of the connections — i.e. one proxy instance or relay replica draining.

Response headers confirm two middleboxes in the path:

```
server: cloudflare
cf-ray: …-MAA
x-envoy-upstream-service-time: …
```

An evening cluster showed one cohort of 8 sockets (all born within 6 seconds) being picked off in singles and pairs over 32 minutes at ages 165–197m — consistent with rolling drains across instances, not a fixed connection TTL.

### Ruled out client-side (with data)

| Hypothesis | Result |
|---|---|
| Relay rate limiting | Refuted. All `rate-limited: quota exceeded` notices occurred in one 64s burst after a mass agent restart; **zero** within hours of any reset. |
| `--relay-observer` traffic | Refuted. The heaviest-traffic window of the day had **zero** resets; a near-idle hour had two. No correlation in either direction. |
| Wi-Fi / link instability | Refuted. `log show` (airportd/configd) shows zero link/roam/DHCP/disassoc events across a 40-minute window in which 8 sockets were reset. |
| Heavy agent activity / large messages | Refuted — inversely correlated if anything. |
| Client ping/pong bug | Refuted. The pong watchdog (`no pong received within 10s`) fired **zero** times on the affected day; these were active resets, not late-detected half-open sockets. |
| Host sleep | Separated out. Sleep-correlated drops are a distinct population with different signatures (`os error 60`, `os error 49`). |

### Impact

Reconnects themselves are reliable (114 successes, 0 retry exhaustions in one day). The cost isn't the reconnect — it's #4752: because resubscribe filters from `now` rather than last-seen, **any mention published during the ~1.4s gap is silently dropped**, with no error surfaced to the user. On a busy multi-agent workspace this is a real, invisible work-loss path.

### Suggested direction

1. Server-side: if connection recycling is intentional (drains, config reloads), send a WebSocket Close frame with a status code so clients can distinguish a planned drain from a network fault.
2. Client-side: make resubscribe resume from **last-seen event** rather than `now` (#4752). That alone would make these blips harmless regardless of what upstream does.
3. Possibly related: #4667 (zombie in-flight turn after reconnect + rate-limit), #3974 (no idle-read watchdog for half-open sockets).

Happy to share the exact relay hostname, `cf-ray` values and full timestamped logs privately if that helps correlate against server-side records.

Contributor guide

Open the contributing guide

Research direction

Start with #4752, then trace the hosted relay WebSocket reconnect and resubscribe path; the issue provides no repository file or test names. Compare reset timestamps and close/error handling against the listed hypotheses. Done requires an agreed server-side close behavior or recovery change that prevents mentions during reconnect gaps from being lost.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, distributed-systems, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.