HarperFast / HarperFast/harper-pro

Replication: table created by replication is treated as locally schema-defined, so later attributes from the source are rejected (error per attribute per message)

Open
#747 0 comments 0 reactions 0 assignees View on GitHub
area:replication bug
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

`ensureTableIfChanged` in `replication/replicationConnection.ts` refuses to add or change attributes whenever the local table has `schemaDefined` set:

```js
if (!existingAttribute || existingAttribute.type !== ensureAttribute.type) {
if (wasSchemaDefined) {
// if the schema is defined, we will not change, we will honor our local definition …
logger.error?.(`Schema for '${databaseName}.${tableDefinition.table}' is defined locally, but attribute '…' from '${remoteNodeName}' does not match local attribute ${existingAttribute ? … : 'which does not exist'}`);
} else { … add / replace the attribute … }
```

The guard is meant to stop two nodes with *different component-declared schemas* from fighting. But `schemaDefined` is also set on tables that were **created on this node by replication** from a peer that has the component — this node never declared anything. Once that happens the table is frozen: every later attribute the peer adds is rejected, and the schema on the node without the component never converges.

Observed on harper-pro 5.2.3 on a 2-node cluster where a component had only been deployed to one node (see HarperFast/harper-pro#746). The node without the component logged, per replicated message, one `[error]` per attribute:

```
[http/2] [error] [replication]: Schema for 'oauth.harper_oauth_mcp_keys' is defined locally, but attribute 'alg: String' from '' does not match local attribute which does not exist
[http/2] [error] [replication]: Schema for 'oauth.harper_oauth_mcp_keys' is defined locally, but attribute 'public_key_pem: String' from '' does not match local attribute which does not exist
[http/2] [error] [replication]: Schema for 'oauth.harper_oauth_mcp_keys' is defined locally, but attribute 'private_key_pem: String' from '' does not match local attribute which does not exist
[http/2] [error] [replication]: Schema for 'oauth.harper_oauth_mcp_keys' is defined locally, but attribute 'created_at: Float' from '' does not match local attribute which does not exist
[http/2] [error] [replication]: Schema for 'flair.Credential' is defined locally, but attribute 'principalId: String' from '' does not match local attribute which does not exist
… (17 attributes of that table, repeated on every reconnect)
```

For `oauth.harper_oauth_mcp_keys` *every* attribute is "which does not exist" — the local definition has none of them, yet it is treated as authoritative. Unchanged on `origin/main`.

## Impact

- Nodes that receive a table via replication before (or without) having the component end up with a `schemaDefined` table whose attributes never track the source. Records still land, but attributes declared later are untyped/unindexed there, and any query relying on the index returns different results per node.
- Error-level log spam on every reconnect (one line per attribute per table), which masks real errors.
- "defined locally" in the message is misleading when the definition came from the peer.

## Proposed behaviour

- Track *where* a table's schema came from. If it was created by replication (not by a local component/`create_table` with a schema), don't treat it as locally authoritative: apply additive changes (new attributes) and log conflicts (type changes) at most once per table, not per message.
- Alternatively, only honor the local definition when `!existingAttribute` is false — a missing local attribute is an addition, not a conflict, and additions can't cause the "battle" the comment worries about.
- Downgrade the repeated message to `warn` and dedupe it per table/attribute.

## Repro

1. Two-node cluster. Deploy a component with a GraphQL-declared schema to node B only (`replicated=false`), write a record so the table replicates to A.
2. Add an attribute to the schema on B and redeploy to B only; write another record.
3. A logs the error above for the new attribute on every message and `describe_table` on A never shows it.

Contributor guide

Open the contributing guide

Research direction

Start in replication/replicationConnection.ts at ensureTableIfChanged and trace how schemaDefined is set when a table arrives from a peer. Reproduce the two-node sequence in the issue, including adding an attribute after the initial replication. Done means replicated tables converge with later source attributes and reconnects no longer emit one error per attribute.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.