HarperFast / HarperFast/harper

Don't park a secondary index on a transient backfill error — retry like the write path

Open
#1,356 0 comments 0 reactions 1 assignee Claimed by @heskew View on GitHub
area:search area:storage enhancement
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

Part of #1354

## Problem
During an index backfill, a single failed record sets `hadIndexingErrors` and leaves the index **parked** (`indexingFailed = true`, `isIndexing` stays true), so the attribute returns 503 until a **process restart** re-triggers the backfill. A momentary, transient storage error (e.g. `ERR_BUSY` / `ERR_TRY_AGAIN` under write pressure) on one record out of many therefore takes the whole index offline indefinitely.

## Evidence
- Park paths: `resources/databases.ts:1530-1553` and outer catch `:1575-1595`
- Failure surfaces: `:1478-1485` (sync), `:1487-1495` (async put rejection)
- Resumable checkpoint: `:1499-1505`
- A proven bounded-backoff retry for exactly these error codes already exists on the write path: `resources/DatabaseTransaction.ts:355-393`

## Proposed change
Classify transient errors (`ERR_BUSY` / `ERR_TRY_AGAIN`) and retry the per-record put with bounded backoff (mirroring `DatabaseTransaction`) before counting it as a failure. Only set `hadIndexingErrors` / park on a **non-transient** error or an exhausted retry budget. Keep park-on-permanent-failure (partial-result protection) intact. Log `warn` while retrying, `error` only when actually parking.

## Compatibility / risk
Strict superset of today's behavior; no descriptor change. Keep retries on the per-record promise so the existing `outstanding` backpressure (`:1507`) still applies. (Note: a readiness gate that awaits the backfill depends on this being correct, since `Table.indexingOperation` resolves even on failure.)

## Acceptance
- A backfill that hits transient errors on some records completes (self-heals) without a restart.
- A genuinely permanent error still parks the index (no silent partial results).

---

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.