HarperFast / HarperFast/harper
Optionally keep a node out of rotation (or gate component load) until secondary indexes are built
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
Part of #1354
> **Design discussion** — this presents options and is open for maintainer input; it is not a single prescribed change.
## Problem
A worker begins serving as soon as components load; index backfills (`runIndexing`) run in the background, so a query against a still-building index returns `503 "not indexed yet"` (`resources/search.ts:335-336`). An app that relies on a secondary index immediately at startup — e.g. a bulk import that dedups with `search()` before `create()` — can run against an incomplete index during that window. Harper has no built-in, opt-in way to say "don't take traffic until my indexes are ready."
## Options (both opt-in, default-off)
### A — Node availability gate (preferred where a traffic director fronts the node)
Node out-of-rotation already exists today via the `availability` status → `set_status` → GTM/load-balancer path (`server/status/definitions.ts:13-14`, `utility/hdbTerms.ts:300`). **What's new is auto-driving that availability from index-build state** ("smarter availability"): report **Unavailable** while a declared index is (re)building so the director routes new traffic — reads *and* writes/imports — to ready peers, then flip to **Available** on completion. Fleet-safe: one node out at a time during a rolling rebuild keeps the service up and sends imports to a node whose index is built → no 503/skip. Needs a **quorum guard** (never eject the last/majority simultaneously) and anti-flap. This variant can also re-apply on a *runtime* reindex (self-eject → rebuild → rejoin) under the guard.
### B — Load-time listen gate (opt-in; for deployments with nothing in front, or the simplest form)
Make component load await `Table.indexingOperation` for the dependent table(s) before the worker reports ready / listens (the listen socket already gates on load completion — `server/threads/threadServer.js:166-203`, listen `:190`, `CHILD_STARTED :201`; await hook at `components/componentLoader.ts:251` or `server/loadRootComponents.js:37-41`). **Load-time only** — cannot cover reindexes after the worker is serving.
## Surface
Per-table directive (e.g. `waitForIndexOnLoad`) and/or a component-level readiness flag; a configurable **timeout** (reuse the `handleApplication` timeout pattern, `components/componentLoader.ts:223-265`) so it logs-and-proceeds rather than hanging forever.
## Sharp edges
- `Table.indexingOperation` **resolves even when the backfill failed** — `runIndexing` swallows errors and leaves `isIndexing=true`/`indexingFailed` (`resources/databases.ts:1530-1547,1575-1595`). So awaiting it does not prove completeness; the gate must re-check `isIndexing`/`indexingFailed` and decide (stay-out vs warn-and-serve).
- **Depends on the don't-park fix** (sibling issue): otherwise a transient backfill error parks the index and the gate holds the node out indefinitely.
## Scope / not a silver bullet
Variant B is load-time only and does **not** cover runtime reindexes (added attribute, schema change, crash-recovery after serving) — those still need the typed retryable 503 (sibling issue). This is a **complement**, not a replacement.
## Compatibility
Opt-in, default-off → behavior unchanged when unset (`indexingOperation` stays fire-and-forget today, `resources/databases.ts:1368`). Additive.
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Assessment
This issue has not been assessed yet.