HarperFast / HarperFast/rocksdb-js
perf: batch the transaction-log iterator for bulk replication catch-up (allocation/call-volume bound)
- Dominant language
- C++
- Stars
- 21
- Forks
- 2
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 36
Description
## Summary
Profiling a production node while it served a **far-behind peer's replication catch-up** shows the transaction-log iterator's `next()` (`src/transaction-log-reader.ts:125`) dominating CPU. The cost is **per-entry allocation churn + call volume**, not the byte reads — so bulk transaction-log streaming (replication catch-up of a large range) is iterator-bound and a good target for a batched read path.
## Evidence
**Profile A — bulk catch-up (node sourcing a peer that resync'd from beyond retention), one busy `http` worker, 6s / 32,415 samples:**
| % | function |
|---|---|
| **86.6%** | `next` @ `rocksdb-js/dist/index.cjs:2056` (transaction-log iterator) |
| 5.0% | (garbage collector) |
| 1.3% | `next` @ harper `replayLogsGuards.js` |
| 0.7% | `getLogMemoryMap` @ rocksdb-js |
| <0.2% ea | RecordEncoder.decode, msgpackr unpack/readMap, auditStore.readAuditEntry, … |
**Profile B — live steady-state (caught-up node, in service), same worker type, 20s:**
| % | function |
|---|---|
| **98.9%** | (idle) |
| ~1% total | `onWSMessage` (replication receive), msgpackr decode, RecordEncoder, `Table` commit/`withLocalEntry`, ordered-binary `writeKey`, analytics `recordAction` |
i.e. the bulk iterator is **absent** from the live path — live replication is apply-bound and cheap when caught up. The iterator cost is specific to **bulk catch-up on the send/source side**.
## Where the cost actually is
In `next()` (`src/transaction-log-reader.ts`), per yielded entry (~lines 220–227):
```js
return { done: false, value: { timestamp, endTxn: Boolean(...), data: logBuffer.subarray(entryStart, position) } };
```
That's **two object literals + a `subarray` view allocated per entry**, returned one entry per call. Streaming a multi-GB log range = *millions* of `next()` calls, each allocating ~3 short-lived objects → the 86.6% self-time is dominated by allocation + iterator-protocol overhead, and feeds the 5% GC. (The `getFloat64`/`getUint32` header reads are ~ns each and are **not** the bottleneck — noting this to correct an initial impression.)
## Optimization directions (highest leverage first)
1. **Batch the iteration for bulk reads.** Entries are contiguous in the mmap'd log buffer, so a bulk API could hand back a large contiguous byte range covering many entries (plus a count / offsets) per call instead of one `{value}` per `next()`. This amortizes the per-call + allocation cost and is the big win for catch-up throughput. The replication send path would frame/split in bulk (or ship count+blob and let the receiver split).
2. **Cut/reuse per-entry allocations** — avoid the per-entry result+value object and `subarray` churn (drives the 5% GC); e.g. yield offsets into the shared buffer, or reuse a result holder.
3. **Skip-path is pure overhead** — when scanning past non-matching entries, the per-entry work (and any allocation) is wasted; a batched/offset approach sidesteps it.
## Caveats
- Single 6s sample of one scenario (sourcing a far-behind peer's bulk catch-up); the live path (Profile B) is ~idle and apply-bound, so **don't generalize #1 to all replication** — it targets bulk catch-up specifically.
- The batched-read API has a wire-protocol + receiver-side design (harper-pro replication send/recv) to think through — not a one-liner.
- Related at the "reduce the need" level: keeping peers within retention avoids beyond-retention base-copies that trigger these large catch-up scans (adjacent to harper-pro#399).
Profiled on a production cluster during an incident recovery. Investigation by Claude (Opus 4.8).
Contributor guide
Assessment
This issue has not been assessed yet.