HarperFast / HarperFast/harper

Investigate ~1 Gbps per-thread MQTT ingest ceiling (single-connection bottleneck)

Open
#662 0 comments 0 reactions 1 assignee Claimed by @kylebernhardy View on GitHub
area:performance bug from-jira
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

**Investigative.** A customer reported MQTT message delays and applied backpressure when a single publishing connection's ingest reached ~1 Gbps. Switching publishing across multiple threads worked around the ceiling. We don't have a confirmed explanation for the per-thread limit.

## What's been ruled out

- No OS / kernel network limit on the host (Akamai support investigated).
- No Linode platform limit identified.
- No Docker network limit identified.
- `iperf` testing on the same machines reached throughput well above 1 Gbps with a single thread.
- Customer publisher has no rate cap; messages are MessagePack-serialized and zip-compressed.

## Hypothesis

The bottleneck is *inside Harper*, per-thread, on the MQTT ingest path:
- Decompression / serialization handling cost per message.
- Event-loop saturation on the receiving thread (a single Node.js worker handling all the parsing + persistence for one connection).
- WS/MQTT framing or per-connection buffering caps.

## Asks

- Reproduce on a staging instance matching the customer's specs.
- Identify the limiting resource on the MQTT receive path (CPU on the receiving thread? buffer drain? sync write to LMDB/RocksDB?).
- If the limit is structural (single-thread per connection), document and provide guidance on connection sharding. If it's tunable, expose the knob.

## Acceptance criteria

- Root cause identified.
- Either: limit raised (single-connection ingest ≥ 2 Gbps on representative hardware), OR documented as a known per-connection ceiling with operator guidance on splitting across connections.

## Investigation pointers (from the comment trail)

- Same instance specs as the customer's prod for reproduction.
- Sample payloads are MessagePack + zip — keep that in any synthetic test.
- A staging instance was offered for the test.

---

🤖 Filed by Claude on behalf of Kris.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.