HarperFast / HarperFast/harper-pro

A readable torn transaction-log frame poisons the replication stream — resync recovers framing, then the undecodable entry head-of-line blocks it

Open
#669 0 comments 0 reactions 0 assignees View on GitHub
area:replication bug
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

The mid-log resync fixes ([harper#2087](https://github.com/HarperFast/harper/pull/2087) + [rocksdb-js#750](https://github.com/HarperFast/rocksdb-js/pull/750)) recover *framing* past a torn transaction-log frame. They do not help when the torn frame is still **readable**, and that is the shape a real partial append usually leaves. The reader yields it as a well-formed entry whose payload is the torn bytes plus its neighbour's, the receiver fails to decode it, closes the connection, resumes from the last durable cursor, and hits the same entry again.

The result is a head-of-line block indistinguishable from the one #2063 was filed on — same stall, same silent `connected: true`, different layer.

## Why the readable shape is the common one

A partial `ENOSPC`/`EDQUOT` append lands the 13-byte header plus M of its declared L payload bytes. The process survives and `O_APPEND` writes the next entry immediately after those M bytes. A reader at the torn frame therefore computes `position + 13 + L`, which is still **inside the file**, so the frame reads as complete — its payload is M real bytes followed by whatever the next entries wrote. Only when L overruns the whole file (a large torn entry, or little appended after) is the frame unreadable and skippable.

Nothing at the reader layer can distinguish this: the header is intact and the length fits. Transaction-log entries carry no per-entry checksum.

## Measured, both engines, same two-node harness

Test: `integrationTests/cluster/txnlogTearReplication.test.mjs` (harper-pro), 60 rows, B's resume cursor established at row 10, tear placed 20 frames from the end of A's `data/local` log.

| Engine | Tear shape | Rows on B | Outcome |
| --- | --- | --- | --- |
| rocksdb-js 2.7.0 (released) | unreadable | 39 / 60 | stops at the frame — the #2016/#2063 defect |
| rocksdb-js #750 build | unreadable | **60 / 60** | resyncs, nothing lost — the fix works |
| rocksdb-js #750 build | **readable** | **39 / 60** | resyncs framing, then wedges on the poison entry |

The third row is this issue. On B, repeatedly:

```
[error] [replication]: Error handling incoming replication message; closing so replication
resumes from the last durable cursor
Error: Data read, but end of buffer not reached {"id":"r39","payload":"xxxx..."}
at checkedRead (msgpackr/unpack.js:217:10)
at Structon.decode (structon/index.js:99:37)
at core/resources/auditStore.ts:629:28
at replication/replicationConnection.ts:4029:28
```

Four such closes before the test's 90 s convergence window expired; the cursor never advances past the entry, so this does not self-heal.

## Direction

The reader cannot fix this, so it falls to the two ends:

1. **Consumer tolerance.** A record that fails to decode should not be able to park a stream forever. [#545 — fix(replication): classify undecodable records + hold on unknown table id (#537)](https://github.com/HarperFast/harper-pro/pull/545) is adjacent and may be the right home, but it was written for a different cause (structure-id divergence) and I have not confirmed it covers a torn payload — worth checking before assuming it does.
2. **Prevention.** [rocksdb-js#748 — A failed transaction-log append orphans its partial bytes](https://github.com/HarperFast/rocksdb-js/issues/748) stops the frame being created at all, which is the only route that removes the poison entry rather than surviving it.

A per-entry checksum would make the readable shape detectable at the reader, turning it back into the skippable case — a larger format change, noted as an option rather than a recommendation.

## Related

- [harper#2016 — Mid-log corrupt transaction-log frame silently truncates replay and replication](https://github.com/HarperFast/harper/issues/2016)
- [harper#2063 — Corrupt-log containment in RocksTransactionLogStore.getRange is per-drain](https://github.com/HarperFast/harper/issues/2063)
- [harper-pro#667 — cluster_status reports a stream healthy after it has lost transaction-log entries](https://github.com/HarperFast/harper-pro/issues/667)

---

🤖 Filed by Claude Opus 5 on behalf of @kriszyp

Contributor guide

Open the contributing guide

Research direction

Start with integrationTests/cluster/txnlogTearReplication.test.mjs and reproduce the readable-tear case. Then inspect replication/replicationConnection.ts around line 4029, core/resources/auditStore.ts around line 629, and adjacent PR #545; done means a torn readable entry no longer permanently blocks replication convergence.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, typescript
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.