HarperFast / HarperFast/harper
Boot transaction-log replay stages one unbounded in-memory batch per version → OOM, unbootable node
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
On startup after an unclean shutdown, transaction-log replay commits and re-creates its `DatabaseTransaction` **only at a version boundary**. Every write within a single version stages into that one transaction, with no intra-version size-based flush. A version carrying a very large number of writes (or one very large write) accumulates an unbounded in-memory write batch and exhausts heap. Because the process dies mid-replay, the next boot restarts the same replay and OOMs again — a boot loop with no forward progress and no self-recovery.
The existing replay guards (harper#1266 / harper#1316) abort **only at version boundaries** ("aborting never tears a same-version write batch in half") and are **progress/time-based, not memory-based**, so they do not fire before the heap is exhausted.
## Where (v5.2.1; core at the v5.2.1 pin)
- `resources/replayLogs.ts`: the transaction is committed + recreated only when `lastTimestamp !== version`; within a version, every `save`/`writeUpdate`/`writeDelete` stages into the single `DatabaseTransaction`. There is no size-based mid-version flush.
- The version-boundary abort guards are progress/time-bounded, not memory-bounded.
## Impact
On an internal replication-heavy cluster whose system transaction logs had grown to multiple GB (see #2140 — RocksDB log pruning never re-arms, which is *how* they grew), an unclean shutdown triggered a replay that exhausted heap on startup and the node could not boot. Multiple nodes entered this unbootable state at once. This is the terminal blast-radius mechanism of a broader replication incident: the trigger was #2153 / harper-pro#545, the growth was #2140, and **this** is why the affected nodes could not recover on restart.
## Suggested fix
Bound the in-memory replay batch by **size**, not just by version boundary — e.g. flush (commit + recreate the transaction) when the staged batch exceeds a byte/record threshold. A mid-version flush is safe for **replay** (idempotent, version-deduped) even though it isn't for live writes, so the same-version-batch invariant that blocks the abort guards does not block a replay-time flush. Alternatively, checkpoint replay progress so a killed replay resumes rather than restarts from scratch.
Relates to #2140 (log growth is the precondition) and #2153 / harper-pro#545 (the incident trigger).
---
🤖 Investigated and filed by Claude (Fable) on Nathan's behalf
Contributor guide
Assessment
This issue has not been assessed yet.