HarperFast / HarperFast/harper
hdb_nodes point-lookup decode fails persistently on repeatedly-upgraded clusters — make point-lookup decode like the scan path (durable #352 fix)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
On repeatedly-upgraded, populated clusters, the `system.hdb_nodes` **point lookup** (`primaryStore.get(name)`) fails to decode a peer's row — a msgpackr shared-structure decode failure carried in from the multi-version upgrade history — while the **range/scan** path decodes the same row fine. This is the persistent variant of harper-pro#352 that its current mitigation does not cure.
## Why it matters (the durable fix behind a replication outage)
Today two layers paper over the failed point decode, both recovery nets rather than cures:
- The #352 mitigation (`resolveNodeForAuth`, harper-pro `replication/knownNodes.ts`) returns a name-only `{ name }` stub so the peer still authenticates.
- harper-pro#461 reconstructs a `{ name, replicates: true }` descriptor on the outbound subscription path so subscriptions are still built.
Both assume the decode failure is transient and "self-heals" once the `system` base-copy resync re-encodes the row against local structures. On affected nodes that base-copy **never completes** (it loops on `hdb_analytics` blob transfer — #388/#385 + the receive-watchdog; the watchdog side is addressed by the `REPLICATION_COPYTIMEOUT` work in harper-pro#454), so the row is never re-encoded, the decode never heals, and correctness stays dependent on the stub.
## Durable fix
Make the `hdb_nodes` point lookup decode the same way the scan path does — decode through the table-layer codec / resolve against the correct shared structures — so `hdb_nodes` reads are correct without a stub and without depending on a base-copy resync to heal. This is the real fix #352 called for.
## Evidence / isolation
- Control: a fresh 5.1.8→5.1.9 cluster with no data does **not** reproduce (zero #352 decode warnings, healthy replication). A repeatedly-upgraded, populated cluster **does** — isolating the cause to the upgrade-history + populated-system-db state, not topology or cold-start churn.
- Symptom on an affected node: the #352 "did not decode to a valid node descriptor … see harper-pro#352" warning every ~60s for over an hour, never self-healing.
## Relationships
- harper-pro#352 (this is the persistent variant the mitigation doesn't cover), harper#1307 / #1308 (per-node typed-structure dict divergence root cause), #1163.
- harper-pro#460 / #461 (the deploy-reload replication break — #461's reconstruct is the recovery net this would make unnecessary on the read side).
- harper-pro#454 + harper#1461 (`REPLICATION_COPYTIMEOUT`) — unblocks the base-copy that the current "self-heal" depends on.
Contributor guide
Research direction
Start at the hdb_nodes point-lookup entry point, primaryStore.get(name), and compare its decode path with the range/scan path. Trace the table-layer codec and shared-structure resolution used by the scan path; done means repeatedly-upgraded populated clusters decode the peer row without a name-only stub or base-copy resync. No test file is named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100