HarperFast / HarperFast/harper

Boot transaction-log replay stages one unbounded in-memory batch per version → OOM, unbootable node

Open
#2,161 3 comments 0 reactions 1 assignee Claimed by @kriszyp View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

On startup after an unclean shutdown, transaction-log replay commits and re-creates its `DatabaseTransaction` **only at a version boundary**. Every write within a single version stages into that one transaction, with no intra-version size-based flush. A version carrying a very large number of writes (or one very large write) accumulates an unbounded in-memory write batch and exhausts heap. Because the process dies mid-replay, the next boot restarts the same replay and OOMs again — a boot loop with no forward progress and no self-recovery.

The existing replay guards (harper#1266 / harper#1316) abort **only at version boundaries** ("aborting never tears a same-version write batch in half") and are **progress/time-based, not memory-based**, so they do not fire before the heap is exhausted.

## Where (v5.2.1; core at the v5.2.1 pin)

- `resources/replayLogs.ts`: the transaction is committed + recreated only when `lastTimestamp !== version`; within a version, every `save`/`writeUpdate`/`writeDelete` stages into the single `DatabaseTransaction`. There is no size-based mid-version flush.
- The version-boundary abort guards are progress/time-bounded, not memory-bounded.

## Impact

On an internal replication-heavy cluster whose system transaction logs had grown to multiple GB (see #2140 — RocksDB log pruning never re-arms, which is *how* they grew), an unclean shutdown triggered a replay that exhausted heap on startup and the node could not boot. Multiple nodes entered this unbootable state at once. This is the terminal blast-radius mechanism of a broader replication incident: the trigger was #2153 / harper-pro#545, the growth was #2140, and **this** is why the affected nodes could not recover on restart.

## Suggested fix

Bound the in-memory replay batch by **size**, not just by version boundary — e.g. flush (commit + recreate the transaction) when the staged batch exceeds a byte/record threshold. A mid-version flush is safe for **replay** (idempotent, version-deduped) even though it isn't for live writes, so the same-version-batch invariant that blocks the abort guards does not block a replay-time flush. Alternatively, checkpoint replay progress so a killed replay resumes rather than restarts from scratch.

Relates to #2140 (log growth is the precondition) and #2153 / harper-pro#545 (the incident trigger).

---
🤖 Investigated and filed by Claude (Fable) on Nathan's behalf

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.