HarperFast / HarperFast/harper
Secondary-index rebuilds: readiness, resilience, and explainability
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
When a secondary index is (re)built, Harper backfills it in the background by walking the primary store; queries on that attribute return `503 "… is not indexed yet"` until the backfill completes (`resources/search.ts:335-336`). That window is fine in principle, but several rough edges in the rebuild lifecycle can turn a routine rebuild into **silent data loss** or an **availability hit** for applications.
A representative failure mode: an application performs a bulk import that **deduplicates per row** with `table.search({ attribute })` and, on a miss, `create()`. If that attribute's index is mid-rebuild, the search throws a 503; if the app's error handling treats it like a business "not found"/skip, it can silently drop rows for the entire rebuild window.
This is a tracking issue for a set of focused, mostly-independent improvements (see the sub-issues):
- **Signal** — the 503 is untyped, so callers can't distinguish "retry shortly" from a permanent failure.
- **Resilience** — a single *transient* backfill error parks the whole index until a manual restart.
- **Avoid churn / explainability** — a rebuild can fire when an index definition is only *representationally* different (not semantically changed), and core never logs *why* a rebuild was triggered.
- **Readiness** — there's no built-in, opt-in way to keep a node out of rotation / hold a component not-ready until its indexes are built, so an app can run against a rebuilding index at startup.
Together: an app should never silently lose data because an index was mid-rebuild; a transient error shouldn't take an index offline until a restart; operators should be able to see why a rebuild happened; and a deployment should be able to opt into "don't take traffic until indexes are ready."
The four improvements are tracked as **sub-issues** of this issue. Related, tracked separately: #1359 — dead `restartNumber` reindex trigger.
## Considered, not pursued in core (recorded)
- **Full-scan fallback during a rebuild** (serve the query via a primary-store scan instead of 503): only helps callers that already opted into scans, and for the common per-row dedup loop it is O(rows × table) — a worse failure than the 503. Note for any future attempt: simply removing the throw is unsafe — the query would then route to the *partial* index and return incomplete results (`resources/search.ts:388`); the indexing case must be folded into the existing `skipIndex` gate to force the full-scan branch (`:434`).
- **Unique secondary constraint / atomic insert-if-absent** (to prevent duplicate rows from concurrent dedup-imports): not viable under Harper's last-write-wins, out-of-order replication — a cross-key uniqueness guarantee can't be honored cluster-wide (locally-unique can be violated once a remote write replicates in). The correct, cluster-safe pattern is application-side — a **deterministic primary key (e.g. a hash of the natural key) + upsert** — which needs no core change (worth documenting).
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Assessment
This issue has not been assessed yet.