HarperFast / HarperFast/harper

HNSW vector orphaned by concurrent write during structural reindex (transient, self-heals)

Open
#1,704 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

A vector inserted **during** an HNSW structural reindex (e.g. redeploying a vector index with a changed `M` parameter, which triggers a clear+rebuild) can be silently **orphaned** — the record is stored fine (a direct PK `get()` returns 200) but is unreachable via vector search. The window is transient (self-heals on the record's next write) but real: confirmed via 10 independent fresh-boot trials with a 0.30% hit rate, defects in 8/10 trials.

## Root cause

`resources/Table.ts:4407-4416`, `updateIndices` calls `index.customIndex.index(...)` **unconditionally** — it is not gated on `index.isIndexing`. This races `runIndexing`'s `dbi.clear()` + backfill in `resources/databases.ts`: a write landing in the middle of the clear/rebuild can call `.index()` against the index instance while it's being cleared/rebuilt underneath, and the new vector is dropped from the graph even though the record write itself commits normally.

## Impact

- **Not permanent** — every orphan observed self-healed with a single ordinary no-op `PUT` to its own record (confirmed via a follow-up clean reindex: 0/21 still orphaned after the touch).
- **Silent** — no error, no indication to the writer that the vector didn't make it into the index. A search-driven read path (e.g. "find similar items") will simply never surface the record until something else happens to rewrite it.
- Realistic trigger: any HNSW index parameter change (e.g. tuning `M`) redeployed while the table is under live write load — not an exotic/adversarial scenario.

## Repro

Harper `ece7da476` (v5.1.15). 10 fully independent trials (fresh Harper boot each — no `dataRootDir`/log reuse, so corpus can't grow cross-trial): 1,500 seed dim-16 vectors/trial, redeploy `M ∈ {8,24}` to force a rebuild, 700 concurrent during-rebuild inserts (8 workers), self-query + PK-cross-check oracle at `ef=400` (near-exhaustive, rules out ANN-approximation false negatives as the cause). Settle gated on the actual `"Finished indexing VecTable attributes"` log line (`databases.ts:1643`), not HTTP latency.

Result: **21/7,000 orphaned (0.30%)**, spread across 8 of 10 trials (per-trial counts: 0,1,0,1,0,0,3,3,0,3).

Test file (not yet promoted to the permanent suite): `integrationTests/qa-scratch/qa495-hnsw-orphan-confirm.test.ts` (lineage: QA-469 → QA-469b → QA-494 → QA-495, the last of which is the clean confirmation).

## Suggested fix direction

Gate the live-write `updateIndices` → `index.customIndex.index(...)` call on `index.isIndexing`, matching how the rest of the index lifecycle treats an in-progress rebuild — either queue/defer the write's index update until the rebuild's backfill catches up to it, or have the backfill itself pick up writes that landed during the clear (so nothing needs a second touch to self-heal).

— KrAIs 🤖 (exploratory QA, on Kris's behalf)

Contributor guide

Open the contributing guide

Research direction

Start with resources/Table.ts:4407-4416 and the indexing lifecycle in resources/databases.ts, especially the rebuild completion at line 1643. Run integrationTests/qa-scratch/qa495-hnsw-orphan-confirm.test.ts to reproduce the race under concurrent writes. Done means writes during a structural HNSW rebuild remain reachable by vector search without a follow-up record write.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.