HarperFast / HarperFast/harper

Secondary-index rebuilds: readiness, resilience, and explainability

Open
#1,354 0 comments 0 reactions 1 assignee Claimed by @heskew View on GitHub
area:search area:storage enhancement
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

When a secondary index is (re)built, Harper backfills it in the background by walking the primary store; queries on that attribute return `503 "… is not indexed yet"` until the backfill completes (`resources/search.ts:335-336`). That window is fine in principle, but several rough edges in the rebuild lifecycle can turn a routine rebuild into **silent data loss** or an **availability hit** for applications.

A representative failure mode: an application performs a bulk import that **deduplicates per row** with `table.search({ attribute })` and, on a miss, `create()`. If that attribute's index is mid-rebuild, the search throws a 503; if the app's error handling treats it like a business "not found"/skip, it can silently drop rows for the entire rebuild window.

This is a tracking issue for a set of focused, mostly-independent improvements (see the sub-issues):

- **Signal** — the 503 is untyped, so callers can't distinguish "retry shortly" from a permanent failure.
- **Resilience** — a single *transient* backfill error parks the whole index until a manual restart.
- **Avoid churn / explainability** — a rebuild can fire when an index definition is only *representationally* different (not semantically changed), and core never logs *why* a rebuild was triggered.
- **Readiness** — there's no built-in, opt-in way to keep a node out of rotation / hold a component not-ready until its indexes are built, so an app can run against a rebuilding index at startup.

Together: an app should never silently lose data because an index was mid-rebuild; a transient error shouldn't take an index offline until a restart; operators should be able to see why a rebuild happened; and a deployment should be able to opt into "don't take traffic until indexes are ready."

The four improvements are tracked as **sub-issues** of this issue. Related, tracked separately: #1359 — dead `restartNumber` reindex trigger.

## Considered, not pursued in core (recorded)

- **Full-scan fallback during a rebuild** (serve the query via a primary-store scan instead of 503): only helps callers that already opted into scans, and for the common per-row dedup loop it is O(rows × table) — a worse failure than the 503. Note for any future attempt: simply removing the throw is unsafe — the query would then route to the *partial* index and return incomplete results (`resources/search.ts:388`); the indexing case must be folded into the existing `skipIndex` gate to force the full-scan branch (`:434`).
- **Unique secondary constraint / atomic insert-if-absent** (to prevent duplicate rows from concurrent dedup-imports): not viable under Harper's last-write-wins, out-of-order replication — a cross-key uniqueness guarantee can't be honored cluster-wide (locally-unique can be violated once a remote write replicates in). The correct, cluster-safe pattern is application-side — a **deterministic primary key (e.g. a hash of the natural key) + upsert** — which needs no core change (worth documenting).

---

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.