HarperFast / HarperFast/harper-pro
Bulk-copy send loop wedges with event-loop starvation on macOS; progresses only via watchdog reconnect cycles
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
On macOS, a local 1 GB `largeClone.test.mjs` run (two loopback nodes, RocksDB) repeatedly wedges the bulk copy: data stops flowing to the clone within seconds of `COPY_START`, the leader's send thread logs the event-loop starvation warning, and progress only resumes when a watchdog forces a reconnect. The copy then completes over several wedge/reconnect cycles (326s for 1 GB ≈ 3 MB/s effective). Observed consistently across 3 runs on macOS 25.5.0 (main @ 160a8eaa). CI's self-hosted Linux runners do not show this signature (their copies run to completion in a single connection).
## Evidence (leader log)
```
14:24:00.904 [http/2] Requesting full copy of database data from ws://127.0.0.27:9933 (no resume cursor for this source)
14:26:00.595 [http/3] Receive watchdog: no ping from 127.0.0.27 (db: "data") for 60000ms — terminating connection and reconnecting — truth={connected: true, state: 2, liveness: 119s ago}
14:26:00.610 [http/2] JavaScript execution has taken too long and is not allowing proper event queue cycling, consider using 'await new Promise(setImmediate)' in code that will execute for a long duration
```
Clone side over the same window: `Sync incomplete; last replication data arrived 126s ago` (arrival stamps frozen), clone data directory not growing. After the reconnect the copy resumes from the cursor and advances again before wedging again.
## Notes
- The `[http/2]` event-loop warning is the thread running the copy send loop for `data` — the send loop appears to hold the event loop long enough to starve its own pings and the reverse connection's liveness (the receive watchdog that fires is for the *other* direction's socket).
- Possibly macOS-specific socket backpressure interacting with the send loop's yield discipline (`waitForDrain` / checkpoint pacing) — 100 KB records, loopback.
- Found while investigating #655 (the sync-monitor fix made the wedge visible: the clone now correctly stays Unavailable and logs stall countdowns instead of declaring success mid-copy).
- Repro: `HARPER_RUN_STRESS_TESTS=1 HARPER_STRESS_LARGE_DATA_GB=1 HARPER_INTEGRATION_TEST_INSTALL_PARENT_DIR=~/dev/tmp/x HARPER_INTEGRATION_TEST_LOG_DIR=~/dev/tmp/x/logs node integrationTests/run.mjs integrationTests/stress/largeClone.test.mjs` on macOS.
Low priority if confirmed macOS-only (stress tests target self-hosted Linux), but the same starvation mechanism under a slow consumer might occur on production Linux under memory/CPU pressure — the send loop yield discipline deserves a look.
Contributor guide
Research direction
Run the supplied HARPER_RUN_STRESS_TESTS command with integrationTests/stress/largeClone.test.mjs on macOS and inspect the bulk-copy send loop, especially waitForDrain and checkpoint pacing. Compare leader and clone logs around the event-loop warning and watchdog reconnects. Done means the 1 GB loopback copy completes without repeated starvation and reconnect cycles, while relevant tests still pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, macos, node.js
- Domain
- backend, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100