HarperFast / HarperFast/rocksdb-js

perf: batch the transaction-log iterator for bulk replication catch-up (allocation/call-volume bound)

Open
#659 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
21
Forks
2
Avg merge
2d 9h
Merged PRs (30d)
36

Description

## Summary

Profiling a production node while it served a **far-behind peer's replication catch-up** shows the transaction-log iterator's `next()` (`src/transaction-log-reader.ts:125`) dominating CPU. The cost is **per-entry allocation churn + call volume**, not the byte reads — so bulk transaction-log streaming (replication catch-up of a large range) is iterator-bound and a good target for a batched read path.

## Evidence

**Profile A — bulk catch-up (node sourcing a peer that resync'd from beyond retention), one busy `http` worker, 6s / 32,415 samples:**

| % | function |
|---|---|
| **86.6%** | `next` @ `rocksdb-js/dist/index.cjs:2056` (transaction-log iterator) |
| 5.0% | (garbage collector) |
| 1.3% | `next` @ harper `replayLogsGuards.js` |
| 0.7% | `getLogMemoryMap` @ rocksdb-js |
| <0.2% ea | RecordEncoder.decode, msgpackr unpack/readMap, auditStore.readAuditEntry, … |

**Profile B — live steady-state (caught-up node, in service), same worker type, 20s:**

| % | function |
|---|---|
| **98.9%** | (idle) |
| ~1% total | `onWSMessage` (replication receive), msgpackr decode, RecordEncoder, `Table` commit/`withLocalEntry`, ordered-binary `writeKey`, analytics `recordAction` |

i.e. the bulk iterator is **absent** from the live path — live replication is apply-bound and cheap when caught up. The iterator cost is specific to **bulk catch-up on the send/source side**.

## Where the cost actually is

In `next()` (`src/transaction-log-reader.ts`), per yielded entry (~lines 220–227):

```js
return { done: false, value: { timestamp, endTxn: Boolean(...), data: logBuffer.subarray(entryStart, position) } };
```

That's **two object literals + a `subarray` view allocated per entry**, returned one entry per call. Streaming a multi-GB log range = *millions* of `next()` calls, each allocating ~3 short-lived objects → the 86.6% self-time is dominated by allocation + iterator-protocol overhead, and feeds the 5% GC. (The `getFloat64`/`getUint32` header reads are ~ns each and are **not** the bottleneck — noting this to correct an initial impression.)

## Optimization directions (highest leverage first)

1. **Batch the iteration for bulk reads.** Entries are contiguous in the mmap'd log buffer, so a bulk API could hand back a large contiguous byte range covering many entries (plus a count / offsets) per call instead of one `{value}` per `next()`. This amortizes the per-call + allocation cost and is the big win for catch-up throughput. The replication send path would frame/split in bulk (or ship count+blob and let the receiver split).
2. **Cut/reuse per-entry allocations** — avoid the per-entry result+value object and `subarray` churn (drives the 5% GC); e.g. yield offsets into the shared buffer, or reuse a result holder.
3. **Skip-path is pure overhead** — when scanning past non-matching entries, the per-entry work (and any allocation) is wasted; a batched/offset approach sidesteps it.

## Caveats

- Single 6s sample of one scenario (sourcing a far-behind peer's bulk catch-up); the live path (Profile B) is ~idle and apply-bound, so **don't generalize #1 to all replication** — it targets bulk catch-up specifically.
- The batched-read API has a wire-protocol + receiver-side design (harper-pro replication send/recv) to think through — not a one-liner.
- Related at the "reduce the need" level: keeping peers within retention avoids beyond-retention base-copies that trigger these large catch-up scans (adjacent to harper-pro#399).

Profiled on a production cluster during an incident recovery. Investigation by Claude (Opus 4.8).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.