HarperFast / HarperFast/harper

Replication receive decoder over-reads on a same-length typed-structure shape fork

Open
#2,151 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

On the replication receive-apply path, if a record's typed-structure **shape at a given id** disagrees with the receiver's handshaked dictionary — but the structure **array length** matches — the record decodes against the wrong shape and over-reads, throwing `Unexpected end of MessagePack data`. Structure change-detection is **length-only** on both the send and save paths, so a same-length shape divergence at an existing id is never detected and never re-sent. Reconnect does not heal it: the sender re-ships its *current* dictionary while replaying immutable log bytes minted against an older dictionary generation. Combined with the close-on-decode-failure policy (HarperFast/harper-pro#521), each miss forces a reconnect that re-hits the same record — a self-amplifying wedge.

Adjacent to but **not covered by** #1163 (closed): that fixed the *local-read* path (a missing structure surfaced instead of silently returning empty). The replication receive decoder is a raw `StructonPackr` with no `_loadStructures` self-heal, and it *over-reads* rather than returning empty, so #1163's guard does not apply here.

## Where (v5.2.1; core at the v5.2.1 pin)

- **Receive decoder** is built only from `TABLE_FIXED_STRUCTURE` frames, rebuilt per handshake, with no `_loadStructures` wired in — `replication/replicationConnection.ts` (the `new StructonPackr({ typedStructs, structures })` construction).
- **Length-only resend gate** — `replication/replicationConnection.ts`: the send side only re-emits `TABLE_FIXED_STRUCTURE` when `typedStructs?.length` / `structures?.length` differ from the last sent lengths.
- **`structureVersion` is itself a length sum** — core `resources/RecordEncoder.ts` (`structures.length + (typedStructs?.length ?? 0)`), and structon's own compatibility check compares only typed/named **lengths**.
- **The over-read** — structon `readStruct`: a present-but-wrong-shape id does **not** throw `Could not find typed structure N`; it reads fixed offsets and variable-field boundaries past the record's real bytes, so the top-level msgpackr unpack hits `position > srcEnd` → `Unexpected end of MessagePack data`.

## Impact

Observed on an internal replication-heavy test cluster: record-bearing operations (`invalidate`/`put`) on churn-heavy structured tables failed to decode on a receiver; with harper-pro#521's close-on-decode-failure this looped (decode → close → resume from the same cursor → same record → repeat), wedging a worker thread. Pure `delete`s were unaffected (they carry no record bit, so they bypass the decoder entirely). HarperFast/harper-pro#545 stops the *loop* (skip instead of close) but does not fix the underlying **divergence** — the record stays undecodable/diverged on that node until a re-clone.

## Distinguishing observable

- `Unexpected end of MessagePack data` (or `Data read, but end of buffer not reached`) ⇒ **shape desync at a present id** (this bug).
- `Could not find typed structure N` ⇒ merely a **missing/lagging id** (#1163 family; heals on reconnect).

The receive-path catch already logs `decoder.typedStructs`, `decoder.structures`, `auditRecord.encoded[:1000]`, and the full `auditRecord` — enough to compare the receiver's shape at the referenced id against the sender's and confirm a same-length mismatch.

## Suggested fix

Detect structure divergence by **shape/content, not array length** — e.g. carry a per-id structure hash (or a content version) in the `TABLE_FIXED_STRUCTURE` exchange and in the save-path `structureVersion`, so a same-length shape change at an existing id triggers a resend. Alternatively, wire a `_loadStructures`/reconciliation into the receive decoder so a shape mismatch re-fetches the authoritative structure instead of over-reading.

The specific cause of the shape fork (cross-thread struct-array reordering vs a reconnect capturing a newer dictionary generation than the replayed bytes) needs a concrete capture to pin down; both windows exist in the code.

---
🤖 Investigated and filed by Claude (Fable) on Nathan's behalf

Contributor guide

Open the contributing guide

Research direction

Start in replication/replicationConnection.ts at the StructonPackr receive construction and the length-only TABLE_FIXED_STRUCTURE resend gate. Then inspect core/resources/RecordEncoder.ts and the receive-path decoder logs to compare same-length structure shapes across sender and receiver generations. Done means a same-length shape divergence is detected and reconciled or re-sent without the decoder over-reading the record.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, typescript
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.