HarperFast / HarperFast/harper

Validate per-worker connection distribution vs V8 heap ceiling under MQTT load (clean-exit ~100k hypothesis)

Open
#2,084 0 comments 0 reactions 0 assignees View on GitHub
area:clustering question
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Problem (hypothesis to confirm on a Fabric node — this is an investigation task, not a code change yet)

A production heap snapshot found worker 0 holding 10,624 of a node's 43,589 MQTT connections — **24.4%, i.e. 3.9× an even share** across 16 workers. With `threads.maxHeapMemory: 2000` (per-worker V8 limit) and ~79 KB heap/connection:

```
2000 MB / 79 KB ≈ 25,300 conns per worker
at 24.4% skew ≈ 104k node connections ← matches the observed ~100k ceiling
```

It would also explain the failure mode: Harper has exited cleanly (exit=0, not OOM-killed) — V8 heap exhaustion terminates the process itself.

## What to do

1. Measure per-worker connection distribution under load (established connections per `-9926.sock`) on an affected node.
2. Determine whether the skew is client-IP affinity hashing (`ipAffinity: true` on the 443/8883 UDS upstreams). If the load generator (k6, ~4 egress IPs) is the cause, the arithmetic (24.4% ≈ 1/4) suggests this is a **test-methodology artifact** and real traffic from many IPs would spread evenly — which changes the whole capacity estimate.
3. If skew is real for production traffic shapes, consider connection-count-aware routing or raising `maxHeapMemory` headroom.

---
_Filed by an AI agent (Claude Code) from a subscription-path memory/CPU investigation; production numbers from a 5.1.26 Fabric node under MQTT load test._

Contributor guide

Open the contributing guide

Research direction

Start on an affected Fabric node by measuring established connections for each -9926.sock under MQTT load, using the 443/8883 UDS upstreams and the k6 setup described. Compare the distribution with ipAffinity: true, the four-egress-IP hypothesis, and threads.maxHeapMemory: 2000. Done means establishing whether skew explains the clean exit near 100k connections and documenting whether production traffic would behave differently.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
backend, networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.