HarperFast / HarperFast/harper
Secondary index opened by a schema-change rescan serves partial results during another worker's backfill
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
When a worker learns of a new secondary index only through the schema-change rescan (`initStores` in `resources/databases.ts`, run by `resetDatabases` on a `schema-change` ITC message), it opens the index with no `isIndexing` flag derived from the catalog's `indexingPID`. `search.ts` gates on `index.isIndexing` alone, so that worker serves the index while the declaring worker's backfill is still running: queries return a subset of matching rows with no 503.
## Reproduction (expected)
1. Populated table; multi-worker node (the default).
2. `create_attribute` (or an `@table` re-declaration) adding an attribute with `indexed: true`.
3. While the backfill runs, query on a **different** worker filtering on that attribute.
Expected: 503 / not-ready (as the declaring worker does). Observed: partial result set, silently.
## Notes
- Pre-existing on `main`; surfaced while reviewing HarperFast/harper#2264, whose branch reload takes the same `initStores` path and inherits the behaviour unchanged (not a regression there).
- Silent wrong-results for the duration of the backfill, bounded by backfill time (long on large tables).
- Fix direction: when `initStores` opens an index whose catalog entry carries an `indexingPID` for a live process, mark it `isIndexing` and clear it on `indexing-finished`, mirroring what the declaring thread does.
🤖 Filed by Claude on behalf of Kris
Contributor guide
Research direction
Start in resources/databases.ts at initStores and resetDatabases, then inspect search.ts where isIndexing gates queries. Trace the catalog indexingPID and schema-change/indexing-finished messages across workers. Done means an index opened during another worker's live backfill is reported not ready until indexing-finished, rather than serving partial results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100