hcengineering / hcengineering/platform
Fix ineffective websocket heartbeat timeouts and reconnect stampedes
- Dominant language
- TypeScript
- Stars
- 27.7k
- Forks
- 2.2k
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
Some websocket heartbeat paths appear to be ineffective or overly chatty across browser clients. This was noticed while inspecting repeated `ping` / `pong!` traffic from a local stack, then confirmed against `hcengineering/platform` `develop`.
### Findings
1. `HulypulseClient` and `PresenceClient` clear and recreate a 5 minute ping timeout on every 30 second ping tick. Because the interval is shorter than the timeout, the timeout can be cleared before it fires.
- `packages/hulypulse-client/src/client.ts`: `PING_INTERVAL_MS = 30s`, `PING_TIMEOUT_MS = 5m`, `startPing()` clears the previous timeout each interval.
- `plugins/presence-resources/src/client.ts`: `PING_INTERVAL = 30s`, `PING_TIMEOUT = 5m`, `startPing()` has the same pattern.
2. In `HulypulseClient`, even if the ping timeout callback fires, it only closes when `this.ws?.readyState !== WebSocket.OPEN`. For a dead-but-open socket that stops returning pong, that condition is false, so the reconnect path may not run.
3. There are multiple heartbeat layers per browser session/tab:
- Transactor client sends app-level `ping` every 10 seconds and expects `pong!`.
- Server session manager can also send an app-level `ping` result for idle sessions.
- HulyPulse sends text `ping` from the client while the Rust server also sends websocket protocol ping frames.
This is expected keepalive traffic in small numbers, but it becomes noisy per tab and can stack across multiple sockets.
4. HulyPulse and presence reconnect on a fixed 1 second timer with no backoff or jitter. During a server restart or network incident, many browser tabs can reconnect in lockstep.
### Expected behavior
- A missing pong should eventually close/reconnect a stale websocket even if the socket remains in `OPEN` state.
- Heartbeat ownership should be clear per socket, avoiding redundant client/server ping loops where one direction is enough.
- Reconnect loops should use bounded backoff and jitter to avoid reconnect stampedes.
### Suggested direction
- Track an outstanding ping/deadline instead of clearing a long timeout every short interval.
- Close on timeout when the socket is still open but no pong was observed.
- Pick one heartbeat owner per socket where possible, or document why both directions are required.
- Add jitter/backoff to reconnect timers.
- Add focused tests for no-pong timeout and reconnect behavior.
### Duplicate check
I searched existing issues before filing and did not find a matching report:
- `repo:hcengineering/platform is:issue heartbeat websocket ping pong`
- `repo:hcengineering/platform is:issue HulyPulse ping timeout`
- `repo:hcengineering/platform is:issue "pong!"`
- `repo:hcengineering/platform is:issue "no response from server"`
- `repo:hcengineering/platform is:issue "no ping response from server"`
- `repo:hcengineering/platform is:issue websocket reconnect backoff`
- `repo:hcengineering/platform is:issue presence websocket ping`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading packages/hulypulse-client/src/client.ts and plugins/presence-resources/src/client.ts, focusing on startPing(), timeout handling, and reconnect timers. Trace the related browser heartbeat paths described in the issue before deciding how ownership should be divided. Done means stale open sockets reconnect after a missing pong, reconnect attempts use bounded backoff and jitter, and focused no-pong and reconnect tests cover the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- frontend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100