HarperFast / HarperFast/harper
Validate per-worker connection distribution vs V8 heap ceiling under MQTT load (clean-exit ~100k hypothesis)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Problem (hypothesis to confirm on a Fabric node — this is an investigation task, not a code change yet)
A production heap snapshot found worker 0 holding 10,624 of a node's 43,589 MQTT connections — **24.4%, i.e. 3.9× an even share** across 16 workers. With `threads.maxHeapMemory: 2000` (per-worker V8 limit) and ~79 KB heap/connection:
```
2000 MB / 79 KB ≈ 25,300 conns per worker
at 24.4% skew ≈ 104k node connections ← matches the observed ~100k ceiling
```
It would also explain the failure mode: Harper has exited cleanly (exit=0, not OOM-killed) — V8 heap exhaustion terminates the process itself.
## What to do
1. Measure per-worker connection distribution under load (established connections per `-9926.sock`) on an affected node.
2. Determine whether the skew is client-IP affinity hashing (`ipAffinity: true` on the 443/8883 UDS upstreams). If the load generator (k6, ~4 egress IPs) is the cause, the arithmetic (24.4% ≈ 1/4) suggests this is a **test-methodology artifact** and real traffic from many IPs would spread evenly — which changes the whole capacity estimate.
3. If skew is real for production traffic shapes, consider connection-count-aware routing or raising `maxHeapMemory` headroom.
---
_Filed by an AI agent (Claude Code) from a subscription-path memory/CPU investigation; production numbers from a 5.1.26 Fabric node under MQTT load test._
Contributor guide
Research direction
Start on an affected Fabric node by measuring established connections for each -9926.sock under MQTT load, using the 443/8883 UDS upstreams and the k6 setup described. Compare the distribution with ipAffinity: true, the four-egress-IP hypothesis, and threads.maxHeapMemory: 2000. Done means establishing whether skew explains the clean exit near 100k connections and documenting whether production traffic would behave differently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- backend, networking, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100