HarperFast / HarperFast/harper
Schema-change write-verify barrier never runs: put(Symbol.for('write-verify')) throws on every path
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
`server/itc/serverHandlers.js` `syncSchemaMetadata` ends with
```js
await databases[msg.database][msg.table].put(Symbol.for('write-verify'), null);
```
intended as a barrier: the receiving thread acknowledges the `schema-change` broadcast only once its in-flight writes under the old schema are committed, which `runIndexing` (`resources/databases.ts`) relies on before scanning the primary store for the backfill.
It has never run. `Table.put` → `_loadRecord` → `checkValidId` rejects a symbol id with `Invalid primary key type: symbol` (`resources/Table.ts`), the handler's `catch` logs it, and the thread acks immediately. Verified empirically on main:
```
put(Symbol) threw: Invalid primary key type: symbol
```
## Consequences
- Every `schema-change` broadcast logs an `Invalid primary key type: symbol` error on every receiving thread (hygiene).
- The intended guarantee is absent: a write in flight on another worker when an index is added can commit after the backfill scanned past its key, leaving one row missing from the new index. Narrow window (a put racing the exact moment of the broadcast), silent.
## Fix direction
Give the table a real barrier method (e.g. await the primary store's pending write queue / a transaction commit) and call that from the handler instead of the symbol put. Found during review of HarperFast/harper#2264, where the branch path deliberately does not copy the idiom. Related: HarperFast/harper#2520 (receiving-thread `isIndexing` gap, the wider version of the same backfill-visibility problem).
🤖 Filed by Claude on behalf of Kris
Contributor guide
Research direction
Start in server/itc/serverHandlers.js at syncSchemaMetadata, then trace Table.put and _loadRecord in resources/Table.ts and the schema backfill path in resources/databases.ts. Reproduce the Symbol primary-key failure and inspect how pending writes or transaction commits are represented. Done means schema-change handling uses a real barrier, no longer logs the invalid-symbol error, and runIndexing cannot scan past an in-flight old-schema write.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, typescript
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100