HarperFast / HarperFast/harper
Investigate ~1 Gbps per-thread MQTT ingest ceiling (single-connection bottleneck)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
**Investigative.** A customer reported MQTT message delays and applied backpressure when a single publishing connection's ingest reached ~1 Gbps. Switching publishing across multiple threads worked around the ceiling. We don't have a confirmed explanation for the per-thread limit.
## What's been ruled out
- No OS / kernel network limit on the host (Akamai support investigated).
- No Linode platform limit identified.
- No Docker network limit identified.
- `iperf` testing on the same machines reached throughput well above 1 Gbps with a single thread.
- Customer publisher has no rate cap; messages are MessagePack-serialized and zip-compressed.
## Hypothesis
The bottleneck is *inside Harper*, per-thread, on the MQTT ingest path:
- Decompression / serialization handling cost per message.
- Event-loop saturation on the receiving thread (a single Node.js worker handling all the parsing + persistence for one connection).
- WS/MQTT framing or per-connection buffering caps.
## Asks
- Reproduce on a staging instance matching the customer's specs.
- Identify the limiting resource on the MQTT receive path (CPU on the receiving thread? buffer drain? sync write to LMDB/RocksDB?).
- If the limit is structural (single-thread per connection), document and provide guidance on connection sharding. If it's tunable, expose the knob.
## Acceptance criteria
- Root cause identified.
- Either: limit raised (single-connection ingest ≥ 2 Gbps on representative hardware), OR documented as a known per-connection ceiling with operator guidance on splitting across connections.
## Investigation pointers (from the comment trail)
- Same instance specs as the customer's prod for reproduction.
- Sample payloads are MessagePack + zip — keep that in any synthetic test.
- A staging instance was offered for the test.
---
🤖 Filed by Claude on behalf of Kris.
Contributor guide
Assessment
This issue has not been assessed yet.