HarperFast / HarperFast/harper
Replication receive decoder over-reads on a same-length typed-structure shape fork
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
On the replication receive-apply path, if a record's typed-structure **shape at a given id** disagrees with the receiver's handshaked dictionary — but the structure **array length** matches — the record decodes against the wrong shape and over-reads, throwing `Unexpected end of MessagePack data`. Structure change-detection is **length-only** on both the send and save paths, so a same-length shape divergence at an existing id is never detected and never re-sent. Reconnect does not heal it: the sender re-ships its *current* dictionary while replaying immutable log bytes minted against an older dictionary generation. Combined with the close-on-decode-failure policy (HarperFast/harper-pro#521), each miss forces a reconnect that re-hits the same record — a self-amplifying wedge.
Adjacent to but **not covered by** #1163 (closed): that fixed the *local-read* path (a missing structure surfaced instead of silently returning empty). The replication receive decoder is a raw `StructonPackr` with no `_loadStructures` self-heal, and it *over-reads* rather than returning empty, so #1163's guard does not apply here.
## Where (v5.2.1; core at the v5.2.1 pin)
- **Receive decoder** is built only from `TABLE_FIXED_STRUCTURE` frames, rebuilt per handshake, with no `_loadStructures` wired in — `replication/replicationConnection.ts` (the `new StructonPackr({ typedStructs, structures })` construction).
- **Length-only resend gate** — `replication/replicationConnection.ts`: the send side only re-emits `TABLE_FIXED_STRUCTURE` when `typedStructs?.length` / `structures?.length` differ from the last sent lengths.
- **`structureVersion` is itself a length sum** — core `resources/RecordEncoder.ts` (`structures.length + (typedStructs?.length ?? 0)`), and structon's own compatibility check compares only typed/named **lengths**.
- **The over-read** — structon `readStruct`: a present-but-wrong-shape id does **not** throw `Could not find typed structure N`; it reads fixed offsets and variable-field boundaries past the record's real bytes, so the top-level msgpackr unpack hits `position > srcEnd` → `Unexpected end of MessagePack data`.
## Impact
Observed on an internal replication-heavy test cluster: record-bearing operations (`invalidate`/`put`) on churn-heavy structured tables failed to decode on a receiver; with harper-pro#521's close-on-decode-failure this looped (decode → close → resume from the same cursor → same record → repeat), wedging a worker thread. Pure `delete`s were unaffected (they carry no record bit, so they bypass the decoder entirely). HarperFast/harper-pro#545 stops the *loop* (skip instead of close) but does not fix the underlying **divergence** — the record stays undecodable/diverged on that node until a re-clone.
## Distinguishing observable
- `Unexpected end of MessagePack data` (or `Data read, but end of buffer not reached`) ⇒ **shape desync at a present id** (this bug).
- `Could not find typed structure N` ⇒ merely a **missing/lagging id** (#1163 family; heals on reconnect).
The receive-path catch already logs `decoder.typedStructs`, `decoder.structures`, `auditRecord.encoded[:1000]`, and the full `auditRecord` — enough to compare the receiver's shape at the referenced id against the sender's and confirm a same-length mismatch.
## Suggested fix
Detect structure divergence by **shape/content, not array length** — e.g. carry a per-id structure hash (or a content version) in the `TABLE_FIXED_STRUCTURE` exchange and in the save-path `structureVersion`, so a same-length shape change at an existing id triggers a resend. Alternatively, wire a `_loadStructures`/reconciliation into the receive decoder so a shape mismatch re-fetches the authoritative structure instead of over-reading.
The specific cause of the shape fork (cross-thread struct-array reordering vs a reconnect capturing a newer dictionary generation than the replayed bytes) needs a concrete capture to pin down; both windows exist in the code.
---
🤖 Investigated and filed by Claude (Fable) on Nathan's behalf
Contributor guide
Research direction
Start in replication/replicationConnection.ts at the StructonPackr receive construction and the length-only TABLE_FIXED_STRUCTURE resend gate. Then inspect core/resources/RecordEncoder.ts and the receive-path decoder logs to compare same-length structure shapes across sender and receiver generations. Done means a same-length shape divergence is detected and reconciled or re-sent without the decoder over-reading the record.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nodejs, typescript
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100