HarperFast / HarperFast/harper

ADD-backfill can produce phantom index entries (F-192, split from #1946)

Open
#2,099 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

Adding an indexed attribute triggers a backfill of the new index over existing records, and that backfill has been observed producing **phantom index entries** — index entries that don't correspond to a live record.

Split out of #1946, which was filed as a single dispatch finding covering two separate defects. #1946 keeps the `drop_attribute` half (F-193); this issue is F-192.

## What is established

- The finding reproduced under a **measured** 4-way worker spread (dispatch task `qa-wave-2026072509`), not an assumed one — the same run that re-confirmed #1946.
- It is one of the two findings underpinning P-472's RED anchor.

## What is not established

Being explicit, because the original finding was a one-line dispatch note and the underlying QA evidence wasn't captured anywhere I can find:

- The exact shape of the phantom entries — stale values, entries for records that were concurrently written, or entries for records that never existed.
- Whether they are reachable by a query (a phantom index entry that returns a row is a correctness bug; one that is skipped on lookup is a space leak).
- Whether the multi-worker precondition is causal or incidental. The backfill racing concurrent writes across workers is the obvious hypothesis, but it is a hypothesis.
- Whether it survives a restart, and whether a reindex clears it.

## Suggested next step

Reproduce with the 4-way worker spread from the original run, then answer the reachability question first — it decides whether this is a correctness issue or a hygiene one, and therefore what priority it should carry. The current P2 assumes hygiene; if a phantom entry can be returned from a query, it should be re-tiered.

## Related

- #1946 — `drop_attribute` never reclaims `Table.indices` (F-193, the sibling finding)
- #1647 — Indexing & query-correctness epic

---

_Filed during backlog triage to split #1946's two findings. No new investigation: everything above the "next step" section comes from the original dispatch note._

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the backfill with the measured 4-way worker spread from dispatch task qa-wave-2026072509. Determine first whether phantom entries are reachable by queries, then investigate restart and reindex behavior; done means the entry shape, concurrency cause, and correctness-versus-hygiene impact are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.