hcengineering / hcengineering/platform

Fix ineffective websocket heartbeat timeouts and reconnect stampedes

Open
#10,946 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
27.7k
Forks
2.2k
PR merge metrics
No merged PRs in 30d

Description

### Summary

Some websocket heartbeat paths appear to be ineffective or overly chatty across browser clients. This was noticed while inspecting repeated `ping` / `pong!` traffic from a local stack, then confirmed against `hcengineering/platform` `develop`.

### Findings

1. `HulypulseClient` and `PresenceClient` clear and recreate a 5 minute ping timeout on every 30 second ping tick. Because the interval is shorter than the timeout, the timeout can be cleared before it fires.
- `packages/hulypulse-client/src/client.ts`: `PING_INTERVAL_MS = 30s`, `PING_TIMEOUT_MS = 5m`, `startPing()` clears the previous timeout each interval.
- `plugins/presence-resources/src/client.ts`: `PING_INTERVAL = 30s`, `PING_TIMEOUT = 5m`, `startPing()` has the same pattern.
2. In `HulypulseClient`, even if the ping timeout callback fires, it only closes when `this.ws?.readyState !== WebSocket.OPEN`. For a dead-but-open socket that stops returning pong, that condition is false, so the reconnect path may not run.
3. There are multiple heartbeat layers per browser session/tab:
- Transactor client sends app-level `ping` every 10 seconds and expects `pong!`.
- Server session manager can also send an app-level `ping` result for idle sessions.
- HulyPulse sends text `ping` from the client while the Rust server also sends websocket protocol ping frames.

This is expected keepalive traffic in small numbers, but it becomes noisy per tab and can stack across multiple sockets.
4. HulyPulse and presence reconnect on a fixed 1 second timer with no backoff or jitter. During a server restart or network incident, many browser tabs can reconnect in lockstep.

### Expected behavior

- A missing pong should eventually close/reconnect a stale websocket even if the socket remains in `OPEN` state.
- Heartbeat ownership should be clear per socket, avoiding redundant client/server ping loops where one direction is enough.
- Reconnect loops should use bounded backoff and jitter to avoid reconnect stampedes.

### Suggested direction

- Track an outstanding ping/deadline instead of clearing a long timeout every short interval.
- Close on timeout when the socket is still open but no pong was observed.
- Pick one heartbeat owner per socket where possible, or document why both directions are required.
- Add jitter/backoff to reconnect timers.
- Add focused tests for no-pong timeout and reconnect behavior.

### Duplicate check

I searched existing issues before filing and did not find a matching report:

- `repo:hcengineering/platform is:issue heartbeat websocket ping pong`
- `repo:hcengineering/platform is:issue HulyPulse ping timeout`
- `repo:hcengineering/platform is:issue "pong!"`
- `repo:hcengineering/platform is:issue "no response from server"`
- `repo:hcengineering/platform is:issue "no ping response from server"`
- `repo:hcengineering/platform is:issue websocket reconnect backoff`
- `repo:hcengineering/platform is:issue presence websocket ping`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading packages/hulypulse-client/src/client.ts and plugins/presence-resources/src/client.ts, focusing on startPing(), timeout handling, and reconnect timers. Trace the related browser heartbeat paths described in the issue before deciding how ownership should be divided. Done means stale open sockets reconnect after a missing pong, reconnect attempts use bounded backoff and jitter, and focused no-pong and reconnect tests cover the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
frontend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.