HarperFast / HarperFast/harper

Migration canonical-structures seed (#1455) is discarded by the typed-table runtime (format mismatch) -> workers mint from empty

Open
#1,509 1 comment 0 reactions 0 assignees View on GitHub
area:replication area:storage bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

`#1455` persists the migration's canonical classic structures as a **bare named array** at `[Symbol.for('structures'), table]` (`bin/copyDb.ts`). But for a **typed table**, the runtime stores/reads the structures as a `Map{named, typed}` — so the runtime's `RecordEncoder.getStructures` does not recognize the bare array and the typed-structure machinery effectively discards it. Result: at runtime start the durable named dictionary is **empty**, and every v5 worker mints the classic structures **from scratch**. #1455's live-fork prevention is therefore a **no-op for typed tables** ("migration tail is 0" only reflects that migrated records self-describe inline).

## Evidence

On every node of a `4.7.34 → 5.1.14` in-place upgrade, the *first* `saveStructures` for the table reads `oldNamedLen=0` (durable empty) despite #1455 having run and the persist key matching the runtime read key (`events/`). A durable dump confirms the runtime location holds a `Map{named,typed}` (worker-rebuilt), and #1455's bare array is not what the runtime reads.

## Impact

Workers minting from an empty durable concurrently is the precondition for the structure-id divergence (#1506 / the replication-apply issue). Seeding one canonical order at startup would largely prevent the divergence rather than relying on recovery.

## Proposed fix

Persist the canonical seed in the **runtime's format** — merge the canonical `named` into the durable `{named, typed}` representation (preserving typed), or seed via the runtime's own structure-persist path — so every worker adopts one canonical order on startup. Note #1455 deliberately drops v4 typed structs; that interacts with this and should be revisited for typed tables.

Refs #1455, #1506, #1453

— filed by KrAIs (Claude) on Kris's behalf

Contributor guide

Open the contributing guide

Research direction

Start with bin/copyDb.ts and the runtime's RecordEncoder.getStructures and saveStructures paths to compare the migration seed format with the typed-table representation. Verify the fix against a 4.7.34 to 5.1.14 upgrade and confirm that the durable named structures are seeded in canonical order while typed structures are preserved and workers do not rebuild from an empty dictionary.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.