HarperFast / HarperFast/rocksdb-js

Committed txnlog reads deliver the unflagged prefix of a transaction torn at a mid-file break

Open
#816 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
21
Forks
2
Avg merge
2d 9h
Merged PRs (30d)
36

Description

## Summary

A committed (`readUncommitted: false`) transaction-log read can deliver the well-framed entries that precede a mid-file framing break even though they belong to a transaction that never committed. The break tore the last entry of that transaction, so its `TRANSACTION_LOG_ENTRY_LAST_FLAG` entry was never written and `Transaction::Commit()` never ran (the log write precedes the RocksDB commit); the earlier entries of the same batch are intact frames and are indistinguishable from committed ones by framing alone.

## Shape

```
[header] e1(flag) e2(flag) e3 e4(torn: length overruns) | resync | batch C (flag) ...
```

`e3` and `e4` are one transaction. `e3` is a complete, unflagged frame; `e4` is torn. Batch C was appended later (legacy writer that kept appending after an ENOSPC short write — the shape #748 retired for new writes). A committed read yields `e1, e2, e3`, then throws `CorruptFrameError { resyncPosition: C }`, then yields `C…`. `e3` is a phantom: nothing in the store ever applied it.

## Where it comes from

- Rotated (non-current) files: pre-existing since #750 — the reader resyncs past a break and the committed bound for a rotated file is its `size`, so the prefix was already delivered.
- The current file: reachable since the scan started advancing `lastCompleteTransactionEnd` past a break (the fix for #815). Before that the watermark stopped before the first break and amputated every committed entry after it — a larger loss, which is why the watermark now trusts the resync heuristic — but the torn transaction's prefix was also excluded as a side effect.

## What a proper fix looks like

A hole-aware committed read: the reader needs to know, per file, the offset where the torn group starts (the first entry after the last flagged entry before the break) and where framing resumes, and throw `CorruptFrameError` at the *group start* rather than at the break in committed mode. That needs:

1. `RecoveryScan` to report the torn group start for each break (not only the first break).
2. Per-file storage of that range on `TransactionLogFile`, populated for rotated files too — today only the current file is scanned at open, and older files only until the watermark seed is found, so this adds a lazy scan on first read of a rotated file with a break.
3. A native accessor the JS reader queries when it maps a file (`getLogMemoryMap`), plus one property compare per entry in the read loop.

Uncommitted reads should keep delivering the prefix: in-flight entries are part of that contract.

## Downstream

On a Harper follower the phantom is a row applied from an entry the source never committed. It is the same class of residue as the torn-DELETE gap accepted for the harper-pro forced base-copy resync: the copy repairs existing keys and does not remove keys that only exist on the follower. The harper-pro follow-up issue for that gap covers this shape as well.

Found by the round-1 cross-model review of the fix for #815.

Contributor guide

Open the contributing guide

Research direction

Start with RecoveryScan and TransactionLogFile, then trace how the current and rotated files are scanned and how getLogMemoryMap feeds the committed read loop. Check the CorruptFrameError and transaction-log read paths; done means committed reads reject the torn group’s prefix while uncommitted reads retain their existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, node.js
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.