HarperFast / HarperFast/harper-pro

Dynamic attribute metadata stops replicating once a table's shared-structure slots are exhausted (peers answer 'unknown attribute' until reconnect)

Open
#707 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

Dynamically-created attribute metadata (attributes born on a peer by an upsert on a dynamic-schema table) replicates to other nodes **only** as a side effect of the record encoder's shared-structure growth: the sender re-sends `TABLE_FIXED_STRUCTURE` (which carries `table.attributes`) exclusively when `encoder.structures.length` / `typedStructs.length` changes ([`replication/replicationConnection.ts` ~4139–4165](https://github.com/HarperFast/harper-pro/blob/main/replication/replicationConnection.ts)).

msgpackr's shared-structure slots default to **32** (`maxSharedStructures`, msgpackr `node.cjs` ~1289). Once a table has minted 32 shared record shapes, further new shapes are encoded inline, `structures.length` plateaus — and **attribute propagation to peers silently stops, permanently**. Records keep replicating fine (inline structures decode correctly), so the remote node stores and can read the record — but any `search_by_id` / `search_by_value` naming the new attribute on the remote node is rejected by `searchValidator` with `unknown attribute 'X'`, indefinitely, until the replication connection happens to reconnect (subscription setup re-sends the full attribute list via `DB_SCHEMA` / `NODE_NAME`).

The remote node also does not index the new attribute's values until it learns the attribute (reconnect → `ensureTableIfChanged` → index backfill), so the exposure window has both a validation-error surface and a stale-index surface.

## Reproduction (deterministic)

3-node cluster (A↔B, A↔C, B↔C), dynamic table `data.qa758` (`create_table` with only `primary_key: id` on each node). Loop `i = 0, 1, 2, …`: upsert on A `{ id: "probe-", ["attr"]: "v" }` with `replicatedConfirmation: 1`, then poll B with `search_by_id … get_attributes: ['id', 'attr']` until it succeeds.

Result (reproduced twice, identically, on main @ `550d2245` v5.2.2, LMDB, Node 26):

- Iterations 0–31: propagate fine (B and C learn each attribute in single-digit ms; measured heal p50 3 ms / p95 7 ms / max 74 ms).
- **Iteration 32 (`attr32` — shared-structure slot #33): never propagates.** After 30+ s of 5 ms polling:
- the record `probe-32` **is present and readable on B** (`search_by_id` on `id` returns it),
- `describe_table` on B and C lists exactly `id, __createdtime__, __updatedtime__, attr0 … attr31` — `attr32` absent on both peers,
- `search_by_id … get_attributes ['id','attr32']` returns HTTP 500 `unknown attribute 'attr32'` on **both** B and C, indefinitely,
- A (the origin) is fully consistent locally.

The stall at exactly 32 matches `maxSharedStructures = 32`: iteration *i* mints shared structure *i + 1* (`{id, attr, __createdtime__, __updatedtime__}`); slot 32 fills at iteration 31.

The sender-side log confirms the mechanism: each healthy iteration logs `send table struct 0 ` (structure count grew → `TABLE_FIXED_STRUCTURE` re-sent, carrying the attribute list); from iteration 32 on, records are sent with no preceding structure message.

## Root cause

Attribute metadata has **no replication path of its own** — it piggybacks on the structure-length gate:

```ts
// replication/replicationConnection.ts (sender, per audit record)
if (typedStructs?.length != tableEntry.typed_length || structures?.length != tableEntry.structure_length) {

ws.send(encode([TABLE_FIXED_STRUCTURE, { typedStructs, structures, attributes: table.attributes, schemaDefined: … }, …]));
}
```

When the structure count stops changing (cap reached — also: any new attribute whose records reuse an existing shape), the gate never fires again. The receiving side (`ensureTableIfChanged`, ~6168) merges attributes correctly whenever it *does* get the message — the defect is purely that the sender stops sending.

Related, same root cause, previously observed by QA exploration (QA-380, 2026-06-26): an explicit `create_attribute` on one node never propagates at all — `addAttributes` → `table()` emits no replication event, and with no subsequent record write minting a new shape, the structure-length gate never fires either.

## Impact

- Long-lived dynamic tables with evolving record shapes (> 32 distinct shapes — easily reached in heterogeneous-ingest workloads) silently stop propagating new attributes cluster-wide; remote nodes serve 500s for queries naming those attributes while holding the data.
- Self-heals only on connection re-establishment, which can be arbitrarily far away on a stable cluster.
- Not data loss (records replicate and are readable via known attributes / `*`), but silent cross-node schema divergence with a wrong-error surface, and remote indexes for the affected attributes lag until reconnect.

## Suggested direction

Give attribute metadata a first-class trigger rather than widening the piggyback:

1. Sender-side: gate the `TABLE_FIXED_STRUCTURE` re-send on `table.schemaVersion` (bumped by `core/resources/databases.ts table()` on any attribute change) *in addition to* structure length. Cheap (integer compare per record), fixes both the cap case and the reused-shape case; the explicit-`create_attribute`-with-no-writes case additionally needs a proactive push (e.g. the sender's schema-change ITC listener marking the table dirty and flushing a `TABLE_FIXED_STRUCTURE` even without a following record).
2. Alternatively (stronger invariant): the receive-apply path could create missing attributes from the records it applies, making "a dynamic table's attribute set covers every record it stores" hold regardless of the metadata channel — but that adds per-record property inspection to the apply hot path, so option 1 is likely the right cost/benefit.

## Discovery context

Found while root-causing the `removeNodeBlastRadius.test.mjs` nightly flake (`unknown attribute 'value'`): the flake itself is a benign first-poll race (fix in the linked PR), but the instrumented probe built to distinguish "record before schema" from "nothing arrived yet" tripped over this permanent stall at iteration 32.

Contributor guide

Open the contributing guide

Research direction

Reproduce the deterministic three-node case from the issue, then read the sender gate in replication/replicationConnection.ts around 4139–4165 and the receiver path ensureTableIfChanged around 6168. Inspect table() in core/resources/databases.ts for schema changes. Done means newly created attributes propagate after shared-structure slots are exhausted, reused shapes and explicit create_attribute are covered, and remote validation and indexes become current without reconnecting.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.