HarperFast / HarperFast/harper-pro
Mixed-version replication: 4.7 receiver throws TypeError on structure ids >= 64 sent by 5.2
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
During a rolling 4.7 → 5.2 upgrade of a two-node replicating pair, the 4.7 node cannot decode records replicated from the 5.2 node once a table's structure dictionary grows past msgpackr's one-byte token range. It fails with a `TypeError` rather than a diagnosable error, and the affected table stops replicating to that node for the duration of the mixed-version window.
```
[error] [replication]: :9933 Error handling incoming replication message
TypeError: currentUnpackr.getStructures is not a function record id:
at Packr.unpack (.../msgpackr/unpack.js:104)
at Packr.decode (.../msgpackr/unpack.js:176)
```
## Environment
- sender: harper-pro 5.2.3, LMDB
- receiver: harperdb 4.7.33, LMDB, msgpackr 1.11.8
- both nodes replicating normally before the upgrade; the upgraded node is the one generating the writes (it holds the app-level leader flag, so it keeps writing while drained of inbound traffic)
- affected table has **1,297 typed structures** accumulated over its lifetime
## Mechanism
msgpackr references structures two ways, and 4.7's decode paths disagree about whether a missing structure is recoverable:
```js
// unpack.js:255 — inline one-byte token (structures 0..63): GUARDED
let structure = currentStructures[token & 0x3f] ||
currentUnpackr.getStructures && loadStructures()[token & 0x3f]
// unpack.js:541 — two-byte reader (structures >= 64): UNGUARDED
let structure = currentStructures[id] || loadStructures()[id]
```
`loadStructures()` calls `currentUnpackr.getStructures()`. Replication decoders are built as
`new Packr({ typedStructs: data.typedStructs, structures: data.structures, ... })` from the
in-band `TABLE_FIXED_STRUCTURE` announcement and never get a `getStructures` provider, so on the
two-byte path the call is a `TypeError` instead of either a fetch or a clear "structure N not
announced" error.
So the failure requires only that the record reference a structure id >= 64 that the receiver
was not sent — which any long-lived, incrementally-patched table will reach.
## Why the announcement doesn't cover it
Both versions re-announce structures only when the encoder's counts change relative to what this
connection last sent (`tableEntry.typed_length` / `structure_length`). 5.2 additionally reloads when
`auditRecord.structureVersion > encoder.typedStructs.length + encoder.structures.length`
(`replication/replicationConnection.ts:4385`); 4.7 has no `structureVersion` concept in its sender or
receiver. `structureVersion` exists in core as far back as v5.1, so this is **not** a 5.2 regression.
## Impact
- the affected table does not replicate to the 4.7 peer while the pair is mixed-version
- the error is thrown per message and caught by the receive loop, which only logs
(`server/replication/replicationConnection.ts` in 4.7), so there is no surfaced failure, no
connection reset, and no operator-visible signal beyond log volume
- whether the un-decodable messages are eventually redelivered or skipped is **not established** —
worth confirming, since it decides whether a mixed-version window risks divergence or only lag
## Suggested direction
Any one of these would remove the sharp edge:
1. Guard the two-byte path the way the one-byte path is guarded, so a missing structure raises a
descriptive error naming the id and table instead of a `TypeError`.
2. Have the 5.x sender re-announce (or fall back to sub-64 / named structures) when the peer is 4.x,
so mixed-version replication cannot reference an unannounced id.
3. Wire a `getStructures` provider onto replication decoders so a missing structure can be fetched.
Happy to test a candidate fix against the same pair — the reproduction is simply a 4.7↔5.2 pair where
the writing node owns a table with >63 accumulated structures.
Contributor guide
Research direction
Reproduce the 4.7↔5.2 pair with a table exceeding 63 accumulated structures, then inspect unpack.js lines 255 and 541 for the differing missing-structure handling. Read replication/replicationConnection.ts around line 4385 and the 4.7 server/replication/replicationConnection.ts receive path; confirm whether failed messages are redelivered or skipped. Done means the mixed-version case no longer produces an unhelpful TypeError and has diagnosable or otherwise safe handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100