HarperFast / HarperFast/harper

hdb_nodes point-lookup decode fails persistently on repeatedly-upgraded clusters — make point-lookup decode like the scan path (durable #352 fix)

Open
#1,463 2 comments 0 reactions 0 assignees View on GitHub
area:replication area:storage bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

On repeatedly-upgraded, populated clusters, the `system.hdb_nodes` **point lookup** (`primaryStore.get(name)`) fails to decode a peer's row — a msgpackr shared-structure decode failure carried in from the multi-version upgrade history — while the **range/scan** path decodes the same row fine. This is the persistent variant of harper-pro#352 that its current mitigation does not cure.

## Why it matters (the durable fix behind a replication outage)

Today two layers paper over the failed point decode, both recovery nets rather than cures:
- The #352 mitigation (`resolveNodeForAuth`, harper-pro `replication/knownNodes.ts`) returns a name-only `{ name }` stub so the peer still authenticates.
- harper-pro#461 reconstructs a `{ name, replicates: true }` descriptor on the outbound subscription path so subscriptions are still built.

Both assume the decode failure is transient and "self-heals" once the `system` base-copy resync re-encodes the row against local structures. On affected nodes that base-copy **never completes** (it loops on `hdb_analytics` blob transfer — #388/#385 + the receive-watchdog; the watchdog side is addressed by the `REPLICATION_COPYTIMEOUT` work in harper-pro#454), so the row is never re-encoded, the decode never heals, and correctness stays dependent on the stub.

## Durable fix

Make the `hdb_nodes` point lookup decode the same way the scan path does — decode through the table-layer codec / resolve against the correct shared structures — so `hdb_nodes` reads are correct without a stub and without depending on a base-copy resync to heal. This is the real fix #352 called for.

## Evidence / isolation

- Control: a fresh 5.1.8→5.1.9 cluster with no data does **not** reproduce (zero #352 decode warnings, healthy replication). A repeatedly-upgraded, populated cluster **does** — isolating the cause to the upgrade-history + populated-system-db state, not topology or cold-start churn.
- Symptom on an affected node: the #352 "did not decode to a valid node descriptor … see harper-pro#352" warning every ~60s for over an hour, never self-healing.

## Relationships

- harper-pro#352 (this is the persistent variant the mitigation doesn't cover), harper#1307 / #1308 (per-node typed-structure dict divergence root cause), #1163.
- harper-pro#460 / #461 (the deploy-reload replication break — #461's reconstruct is the recovery net this would make unnecessary on the read side).
- harper-pro#454 + harper#1461 (`REPLICATION_COPYTIMEOUT`) — unblocks the base-copy that the current "self-heal" depends on.

Contributor guide

Open the contributing guide

Research direction

Start at the hdb_nodes point-lookup entry point, primaryStore.get(name), and compare its decode path with the range/scan path. Trace the table-layer codec and shared-structure resolution used by the scan path; done means repeatedly-upgraded populated clusters decode the peer row without a name-only stub or base-copy resync. No test file is named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.