HarperFast / HarperFast/harper
HNSW vector orphaned by concurrent write during structural reindex (transient, self-heals)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
A vector inserted **during** an HNSW structural reindex (e.g. redeploying a vector index with a changed `M` parameter, which triggers a clear+rebuild) can be silently **orphaned** — the record is stored fine (a direct PK `get()` returns 200) but is unreachable via vector search. The window is transient (self-heals on the record's next write) but real: confirmed via 10 independent fresh-boot trials with a 0.30% hit rate, defects in 8/10 trials.
## Root cause
`resources/Table.ts:4407-4416`, `updateIndices` calls `index.customIndex.index(...)` **unconditionally** — it is not gated on `index.isIndexing`. This races `runIndexing`'s `dbi.clear()` + backfill in `resources/databases.ts`: a write landing in the middle of the clear/rebuild can call `.index()` against the index instance while it's being cleared/rebuilt underneath, and the new vector is dropped from the graph even though the record write itself commits normally.
## Impact
- **Not permanent** — every orphan observed self-healed with a single ordinary no-op `PUT` to its own record (confirmed via a follow-up clean reindex: 0/21 still orphaned after the touch).
- **Silent** — no error, no indication to the writer that the vector didn't make it into the index. A search-driven read path (e.g. "find similar items") will simply never surface the record until something else happens to rewrite it.
- Realistic trigger: any HNSW index parameter change (e.g. tuning `M`) redeployed while the table is under live write load — not an exotic/adversarial scenario.
## Repro
Harper `ece7da476` (v5.1.15). 10 fully independent trials (fresh Harper boot each — no `dataRootDir`/log reuse, so corpus can't grow cross-trial): 1,500 seed dim-16 vectors/trial, redeploy `M ∈ {8,24}` to force a rebuild, 700 concurrent during-rebuild inserts (8 workers), self-query + PK-cross-check oracle at `ef=400` (near-exhaustive, rules out ANN-approximation false negatives as the cause). Settle gated on the actual `"Finished indexing VecTable attributes"` log line (`databases.ts:1643`), not HTTP latency.
Result: **21/7,000 orphaned (0.30%)**, spread across 8 of 10 trials (per-trial counts: 0,1,0,1,0,0,3,3,0,3).
Test file (not yet promoted to the permanent suite): `integrationTests/qa-scratch/qa495-hnsw-orphan-confirm.test.ts` (lineage: QA-469 → QA-469b → QA-494 → QA-495, the last of which is the clean confirmation).
## Suggested fix direction
Gate the live-write `updateIndices` → `index.customIndex.index(...)` call on `index.isIndexing`, matching how the rest of the index lifecycle treats an in-progress rebuild — either queue/defer the write's index update until the rebuild's backfill catches up to it, or have the backfill itself pick up writes that landed during the clear (so nothing needs a second touch to self-heal).
— KrAIs 🤖 (exploratory QA, on Kris's behalf)
Contributor guide
Research direction
Start with resources/Table.ts:4407-4416 and the indexing lifecycle in resources/databases.ts, especially the rebuild completion at line 1643. Run integrationTests/qa-scratch/qa495-hnsw-orphan-confirm.test.ts to reproduce the race under concurrent writes. Done means writes during a structural HNSW rebuild remain reachable by vector search without a follow-up record write.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100