HarperFast / HarperFast/harper-pro

Mixed-version replication: 4.7 receiver throws TypeError on structure ids >= 64 sent by 5.2

Open
#729 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

During a rolling 4.7 → 5.2 upgrade of a two-node replicating pair, the 4.7 node cannot decode records replicated from the 5.2 node once a table's structure dictionary grows past msgpackr's one-byte token range. It fails with a `TypeError` rather than a diagnosable error, and the affected table stops replicating to that node for the duration of the mixed-version window.

```
[error] [replication]: :9933 Error handling incoming replication message
TypeError: currentUnpackr.getStructures is not a function record id:
at Packr.unpack (.../msgpackr/unpack.js:104)
at Packr.decode (.../msgpackr/unpack.js:176)
```

## Environment

- sender: harper-pro 5.2.3, LMDB
- receiver: harperdb 4.7.33, LMDB, msgpackr 1.11.8
- both nodes replicating normally before the upgrade; the upgraded node is the one generating the writes (it holds the app-level leader flag, so it keeps writing while drained of inbound traffic)
- affected table has **1,297 typed structures** accumulated over its lifetime

## Mechanism

msgpackr references structures two ways, and 4.7's decode paths disagree about whether a missing structure is recoverable:

```js
// unpack.js:255 — inline one-byte token (structures 0..63): GUARDED
let structure = currentStructures[token & 0x3f] ||
currentUnpackr.getStructures && loadStructures()[token & 0x3f]

// unpack.js:541 — two-byte reader (structures >= 64): UNGUARDED
let structure = currentStructures[id] || loadStructures()[id]
```

`loadStructures()` calls `currentUnpackr.getStructures()`. Replication decoders are built as
`new Packr({ typedStructs: data.typedStructs, structures: data.structures, ... })` from the
in-band `TABLE_FIXED_STRUCTURE` announcement and never get a `getStructures` provider, so on the
two-byte path the call is a `TypeError` instead of either a fetch or a clear "structure N not
announced" error.

So the failure requires only that the record reference a structure id >= 64 that the receiver
was not sent — which any long-lived, incrementally-patched table will reach.

## Why the announcement doesn't cover it

Both versions re-announce structures only when the encoder's counts change relative to what this
connection last sent (`tableEntry.typed_length` / `structure_length`). 5.2 additionally reloads when
`auditRecord.structureVersion > encoder.typedStructs.length + encoder.structures.length`
(`replication/replicationConnection.ts:4385`); 4.7 has no `structureVersion` concept in its sender or
receiver. `structureVersion` exists in core as far back as v5.1, so this is **not** a 5.2 regression.

## Impact

- the affected table does not replicate to the 4.7 peer while the pair is mixed-version
- the error is thrown per message and caught by the receive loop, which only logs
(`server/replication/replicationConnection.ts` in 4.7), so there is no surfaced failure, no
connection reset, and no operator-visible signal beyond log volume
- whether the un-decodable messages are eventually redelivered or skipped is **not established** —
worth confirming, since it decides whether a mixed-version window risks divergence or only lag

## Suggested direction

Any one of these would remove the sharp edge:

1. Guard the two-byte path the way the one-byte path is guarded, so a missing structure raises a
descriptive error naming the id and table instead of a `TypeError`.
2. Have the 5.x sender re-announce (or fall back to sub-64 / named structures) when the peer is 4.x,
so mixed-version replication cannot reference an unannounced id.
3. Wire a `getStructures` provider onto replication decoders so a missing structure can be fetched.

Happy to test a candidate fix against the same pair — the reproduction is simply a 4.7↔5.2 pair where
the writing node owns a table with >63 accumulated structures.

Contributor guide

Open the contributing guide

Research direction

Reproduce the 4.7↔5.2 pair with a table exceeding 63 accumulated structures, then inspect unpack.js lines 255 and 541 for the differing missing-structure handling. Read replication/replicationConnection.ts around line 4385 and the 4.7 server/replication/replicationConnection.ts receive path; confirm whether failed messages are redelivered or skipped. Done means the mixed-version case no longer produces an unhelpful TypeError and has diagnosable or otherwise safe handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.