HarperFast / HarperFast/harper

Secondary index opened by a schema-change rescan serves partial results during another worker's backfill

Open
#2,520 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

When a worker learns of a new secondary index only through the schema-change rescan (`initStores` in `resources/databases.ts`, run by `resetDatabases` on a `schema-change` ITC message), it opens the index with no `isIndexing` flag derived from the catalog's `indexingPID`. `search.ts` gates on `index.isIndexing` alone, so that worker serves the index while the declaring worker's backfill is still running: queries return a subset of matching rows with no 503.

## Reproduction (expected)

1. Populated table; multi-worker node (the default).
2. `create_attribute` (or an `@table` re-declaration) adding an attribute with `indexed: true`.
3. While the backfill runs, query on a **different** worker filtering on that attribute.

Expected: 503 / not-ready (as the declaring worker does). Observed: partial result set, silently.

## Notes

- Pre-existing on `main`; surfaced while reviewing HarperFast/harper#2264, whose branch reload takes the same `initStores` path and inherits the behaviour unchanged (not a regression there).
- Silent wrong-results for the duration of the backfill, bounded by backfill time (long on large tables).
- Fix direction: when `initStores` opens an index whose catalog entry carries an `indexingPID` for a live process, mark it `isIndexing` and clear it on `indexing-finished`, mirroring what the declaring thread does.

🤖 Filed by Claude on behalf of Kris

Contributor guide

Open the contributing guide

Research direction

Start in resources/databases.ts at initStores and resetDatabases, then inspect search.ts where isIndexing gates queries. Trace the catalog indexingPID and schema-change/indexing-finished messages across workers. Done means an index opened during another worker's live backfill is reported not ready until indexing-finished, rather than serving partial results.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.