HarperFast / HarperFast/harper
Corrupt-log containment in RocksTransactionLogStore.getRange is per-drain, so one bad entry head-of-line-blocks a replication stream forever
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
A corrupt transaction-log entry permanently head-of-line-blocks a replication stream, because the corrupt-entry containment added in `RocksTransactionLogStore.getRange` is scoped to a **single drain**. The `failedIterators` `WeakSet` is constructed *inside* `getRange` and keyed on iterator object identity, so every fresh `getRange` builds new iterators, re-reads the same corrupt log, and re-throws. There is no durable exclusion, no recovery, and no health signal — the stream simply never advances past the bad entry.
Observed on 5.1.x: the same corrupt log re-erroring on a fixed cadence for five days while replication for that database made no forward progress.
## Mechanism
`resources/RocksTransactionLogStore.ts` (`getRange`):
```ts
const failedIterators = new WeakSet>(); // per-call
...
const safeNext = (iterator, log?) => {
if (failedIterators.has(iterator)) return { value: undefined, done: true };
try { ... } catch { failedIterators.add(iterator); ... }
};
```
Within one `getRange` this behaves as intended: the corrupt log is marked failed, subsequent retry-polls skip it, other peer logs keep draining, and the worker stays healthy (the original goal — avoiding an `uncaughtException` out of the aggregate iterator).
Across calls it provides nothing. The `WeakSet` and the iterators both die with the call, so the next drain re-opens the same log, hits the same entry, and logs the same error. Combined with the replay guard — which *stops* iteration at a corrupt entry rather than skipping or excluding the log — nothing after the bad entry is ever delivered.
## Impact
- Replication for the affected database stops advancing permanently. Records written after the corrupt entry never reach the peer.
- The failure is `warn`/`error` log spam only. `cluster_status` still shows the connection `connected: true`; there is no "this stream is wedged" signal, so the condition is invisible to monitoring and can persist for days.
- Recovery today is manual and requires knowing to look at the transaction-log files at all.
## Expected
Corrupt-log exclusion should survive beyond a single `getRange`: track the failed `(log, position)` durably for the store so later drains skip it, and/or advance past the corrupt entry rather than stopping. Either way the condition needs to surface as a health signal — an operator should not have to grep logs to discover that a replication stream has been dead for days.
## Prior discussion
This exact gap was raised in review on the PR that introduced the containment ("the aggregate iterator should also remove/exclude the corrupt log so later drains do not keep retrying it"). That PR was later closed, and only the per-call containment landed via a separate commit, so the durable-exclusion half was never implemented.
## Related
- #1977 — torn audit entry throws `RangeError` out of transaction-log iteration instead of degrading. Complementary: that asks the *audit* reader to degrade the way the *replay* reader already does. This issue is about the replay reader's degradation being non-durable — it stops cleanly, but forever.
- #1135 — corrupt length-prefix entry aborting startup (closed; introduced the stop-at-corrupt-entry behavior).
Filed from a field incident; cluster and host identifiers omitted.
Contributor guide
Research direction
Start in resources/RocksTransactionLogStore.ts at getRange, tracing failedIterators, iterator recreation, and the replay guard across drain calls. Compare the containment with related issues #1977 and #1135 and the prior review context. Done means a corrupt (log, position) no longer wedges later drains and the condition is exposed as a health signal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100