HarperFast / HarperFast/rocksdb-js

Uncommitted transaction-log reads bound corruption checks by mapped capacity, so a torn frame with a plausible length goes undetected

Open
#749 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
21
Forks
2
Avg merge
2d 9h
Merged PRs (30d)
36

Description

## Summary

An uncommitted transaction-log read bounds its per-entry corruption checks by the **mapped capacity** of the memory map rather than by the written extent. The map is pre-extended far past the data, so a torn frame only trips the check when its declared length exceeds that whole capacity. Any smaller garbage length is accepted, and the reader advances `position` by it — walking over valid entries, or handing back a misframed one, without ever reporting corruption.

## Where

`src/transaction-log-reader.ts`, in the `query()` iterator:

```ts
const limit = readUncommitted ? logBuffer!.length : Math.min(size, logBuffer!.length);
if (position + TRANSACTION_LOG_ENTRY_HEADER_SIZE > limit) { /* corrupt */ }
const length = dataView.getUint32(position + 8);
if (position + TRANSACTION_LOG_ENTRY_HEADER_SIZE + length > limit) { /* corrupt */ }
```

For a committed read `limit` is the committed watermark, which is a true entry boundary, so the checks are tight. For `readUncommitted` it is `logBuffer.length` — the mapped capacity.

## Why it matters

`readUncommitted` is the boot-replay path (Harper's `replayLogs.ts` reads with `startFromLastFlushed: true, readUncommitted: true`), which is precisely the reader that has to cope with a log damaged by an interrupted append.

The field error in HarperFast/harper#1977 declared a length of 1,688,494,450 — comfortably past any mapped capacity, so it was caught. A partial write that leaves a plausible-looking smaller length is not: the frame is silently accepted, and iteration continues from a position that is no longer an entry boundary.

This also gates corrupt-frame resync (HarperFast/harper#2016, #2063): recovery can only run where detection fires, so on this path resync is inert for the class of corruption whose declared length happens to land under the mapped capacity.

## Why it is not simply tightened

The looser bound is deliberate — an uncommitted reader is allowed to read past the `size` it captured at query time, because a concurrent append may have grown the file since. Replacing `logBuffer.length` with the extent captured at query start would report healthy, concurrently-appended entries as corrupt.

Fetching the current extent per entry is also not acceptable: `getLogFileSize` crosses into native and takes the store mutex, so it must not sit on the healthy read path (this was measured as a regression in PR review and moved off that path).

## Suggested direction

Check against a cached written extent and refresh it only on an apparent overrun:

1. bound the checks by `min(cachedWrittenExtent, logBuffer.length)`, seeded once per `query()`;
2. when an entry appears to overrun that bound, re-read `getLogFileSize` **once** and re-test before declaring corruption;
3. only report a corrupt frame if it still overruns.

A concurrent append then costs one refresh rather than a false corruption report, the healthy path costs nothing per entry, and detection becomes as tight on the uncommitted path as it already is on the committed one.

## Related

- #748 — the write-side producer of these mid-file breaks
- HarperFast/harper#2016, HarperFast/harper#2063 — the reader-side consequences

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start in src/transaction-log-reader.ts at the query() iterator, then inspect the readUncommitted call from replayLogs.ts and the getLogFileSize path. Implement the cached-extent check with a single refresh on apparent overrun, preserving concurrent appends while detecting smaller torn-frame lengths; done means healthy growth remains readable and corruption is reported when the refreshed extent still fails the bounds check.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.