HarperFast / HarperFast/harper

Schema-change write-verify barrier never runs: put(Symbol.for('write-verify')) throws on every path

Open
#2,522 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

`server/itc/serverHandlers.js` `syncSchemaMetadata` ends with

```js
await databases[msg.database][msg.table].put(Symbol.for('write-verify'), null);
```

intended as a barrier: the receiving thread acknowledges the `schema-change` broadcast only once its in-flight writes under the old schema are committed, which `runIndexing` (`resources/databases.ts`) relies on before scanning the primary store for the backfill.

It has never run. `Table.put` → `_loadRecord` → `checkValidId` rejects a symbol id with `Invalid primary key type: symbol` (`resources/Table.ts`), the handler's `catch` logs it, and the thread acks immediately. Verified empirically on main:

```
put(Symbol) threw: Invalid primary key type: symbol
```

## Consequences

- Every `schema-change` broadcast logs an `Invalid primary key type: symbol` error on every receiving thread (hygiene).
- The intended guarantee is absent: a write in flight on another worker when an index is added can commit after the backfill scanned past its key, leaving one row missing from the new index. Narrow window (a put racing the exact moment of the broadcast), silent.

## Fix direction

Give the table a real barrier method (e.g. await the primary store's pending write queue / a transaction commit) and call that from the handler instead of the symbol put. Found during review of HarperFast/harper#2264, where the branch path deliberately does not copy the idiom. Related: HarperFast/harper#2520 (receiving-thread `isIndexing` gap, the wider version of the same backfill-visibility problem).

🤖 Filed by Claude on behalf of Kris

Contributor guide

Open the contributing guide

Research direction

Start in server/itc/serverHandlers.js at syncSchemaMetadata, then trace Table.put and _loadRecord in resources/Table.ts and the schema backfill path in resources/databases.ts. Reproduce the Symbol primary-key failure and inspect how pending writes or transaction commits are represented. Done means schema-change handling uses a real barrier, no longer logs the invalid-symbol error, and runIndexing cannot scan past an in-flight old-schema write.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, typescript
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.