HarperFast / HarperFast/harper
ADD-backfill can produce phantom index entries (F-192, split from #1946)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
Adding an indexed attribute triggers a backfill of the new index over existing records, and that backfill has been observed producing **phantom index entries** — index entries that don't correspond to a live record.
Split out of #1946, which was filed as a single dispatch finding covering two separate defects. #1946 keeps the `drop_attribute` half (F-193); this issue is F-192.
## What is established
- The finding reproduced under a **measured** 4-way worker spread (dispatch task `qa-wave-2026072509`), not an assumed one — the same run that re-confirmed #1946.
- It is one of the two findings underpinning P-472's RED anchor.
## What is not established
Being explicit, because the original finding was a one-line dispatch note and the underlying QA evidence wasn't captured anywhere I can find:
- The exact shape of the phantom entries — stale values, entries for records that were concurrently written, or entries for records that never existed.
- Whether they are reachable by a query (a phantom index entry that returns a row is a correctness bug; one that is skipped on lookup is a space leak).
- Whether the multi-worker precondition is causal or incidental. The backfill racing concurrent writes across workers is the obvious hypothesis, but it is a hypothesis.
- Whether it survives a restart, and whether a reindex clears it.
## Suggested next step
Reproduce with the 4-way worker spread from the original run, then answer the reachability question first — it decides whether this is a correctness issue or a hygiene one, and therefore what priority it should carry. The current P2 assumes hygiene; if a phantom entry can be returned from a query, it should be re-tiered.
## Related
- #1946 — `drop_attribute` never reclaims `Table.indices` (F-193, the sibling finding)
- #1647 — Indexing & query-correctness epic
---
_Filed during backlog triage to split #1946's two findings. No new investigation: everything above the "next step" section comes from the original dispatch note._
Contributor guide
Research direction
Start by reproducing the backfill with the measured 4-way worker spread from dispatch task qa-wave-2026072509. Determine first whether phantom entries are reachable by queries, then investigate restart and reindex behavior; done means the entry shape, concurrency cause, and correctness-versus-hygiene impact are established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100