HarperFast / HarperFast/harper

restore_backup: on_restart route to replace a component-held database at the next startup

Open
#2,638 0 comments 0 reactions 1 assignee Claimed by @cb1kenobi View on GitHub
area:operations-api area:storage enhancement
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

Part of #2632 (item 6) — the **replace-existing** PR. Design note §5.6, §5.7, §5.8 in `docs/proposals/archive-restore.md` on branch `design/archive-restore`. Depends on #2635, #2636, #2637, #2640, and either #2451 or minting a generation here.

## Problem

`restore_backup` closes the database across worker threads and returns 409 when a handle remains (`verifyDatabaseClosed`, `dataLayer/rocksdbBackup.ts:624`), pointing at the offline CLI. Any database a component declares tables in is component-held, and on Fabric the operator cannot stop the server, so `restore_backup` cannot restore the databases people actually use there. This is independent of archives; it is #1831's Fabric gap.

## What to build

**`on_restart: true`** on `restore_backup`, explicit rather than automatic on 409 (a restart drops availability; the 409 message names the flag). The request validates as today (backup complete, blob roots compatible, no peers, room for the pre-restore backup), writes an **intent file** in state `pending`, inserts the job row, returns the job id, drains workers through the rolling path (`restart` today posts `SHUTDOWN` and terminates immediately — `server/threads/manageThreads.js:983-995`), and restarts.

**The intent file** lives beside the restore marker under the reserved `` `restore` `` directory, written temp → fsync → rename, versioned, and holds: operation id, database, `backup_id`, job id, `retain_previous` and the pinned pre-restore backup id, expected table set, blob-completeness decision, generation, requester, attempt count, and state. It is *not* the marker: `beginRestore()` truncates the marker before rewriting it (#2634). Malformed or torn intent fails closed (name blocked).

| state | discovery of the name | next boot |
|---|---|---|
| `pending` | blocked | run the hook |
| `applying` | blocked (marker too) | run the hook; apply is rerunnable |
| `committed` | open | finish finalization only — never re-apply |
| `terminal` | blocked | boot in safe mode; wait for `resume`/`supersede`/`cancel` |

**The boot hook** runs in `server/loadRootComponents.js` right after `getTables()` and before the root component loads — `system` is open (peer check), the target is unpublished (the intent blocks discovery like `databasesBlockedByRestore`), no worker exists, and no plugin has loaded. Sequence per intent, attempt counter fsynced first:

1. Peers, read by core from `system.hdb_nodes` (rows other than `getThisNodeName()`) and `replication.routes` — no pro predicate, which cannot be registered yet. Peers → `terminal`.
2. Pre-restore backup **exactly once, while `pending`** (`createBackupOffline` refuses a marked database, `:965-969`, so an `applying` retry must reuse the pinned id, never recapture). Taken here, after the drain, it captures the final acknowledged write; durable and pinned before step 3. Skipped for `retain_previous: false` and for a supersede/rollback.
3. State → `applying`. The apply helper (#2637) in the engine child process: lock, marker, purge, engine restore, blob restore — **no `LOCK` probe** at boot (the hook is the server; a probe that aborts on a corrupt old `CURRENT` would otherwise make the database unrecoverable through any `resume`/`supersede`), and no marker clear.
4–7. Inventory check, private open, expected-table check, restore-strict elected replay, blob completeness, durability barriers — the publication protocol from #2637.
8. State → `committed`, fsynced, **before** the discovery block lifts. Then finalize idempotently: clear marker, publish via the ordinary load, reconcile the job row, delete the intent. A crash after `committed` only re-runs finalization.

**Failure never exits.** Under `HARPER_EXIT_ON_RESTART` a restart is `process.exit(0)` and the orchestrator loops (`bin/restart.ts:162-166`); on self-managed installs there is no supervisor at all (`harper start` forks and its restart handler exits with the launcher — `utility/processManagement/processManagement.js:36-78`). A failed attempt records the failure, keeps marker and intent, and continues boot with the database blocked. Retry on the next restart or `resume: true`; after the attempt limit, `terminal`. A `terminal` intent triggers **safe mode by file** for that boot (`HARPER_SAFE_MODE` exists but is env-only — `bin/run.ts:43-44`, `server/loadRootComponents.js:48,61,77-82` — and a container cannot set its own env for the next boot); component load failures are already contained per component (`components/componentLoader.ts:1165-1174`). Document `HARPER_SAFE_MODE`.

**API on a pending or terminal intent**, super-user, working without the database loaded (#2636): `resume=true` re-arms and restarts; `backup_id= on_restart=true` **supersedes** (this is rollback to the pre-restore id — no new pre-restore backup is taken); `cancel=true` removes the intent (unblocks only if `pending`; otherwise the marker keeps the block and the response says so). A `committed` intent can only be finished.

**Retained copy**: `retain_previous` defaults to true; declining is explicit and recorded in the job; the pre-restore backup is a normal managed backup, pinned while an intent names it, pruned by `purge_backups` afterwards. Its cost (one more N in the repository) is real under a quota, which is why it is an operator decision on the record rather than a heuristic.

## Acceptance

- `get_backup` → `import_backup` → `restore_backup on_restart=true` on a component-held database → supervised exit and restart → records and blobs readable; the job row shows the outcome; the pre-restore backup is listed and pinned.
- Kill injection at every step: between pre-restore backup completion and pin (retry reuses the pinned id, never recaptures); between pin and first destructive write; during apply; between `committed` and the job-row write followed by new writes and another restart — nothing re-applies and the marker is gone.
- A corrupt *old* destination at boot is replaced on the next attempt (no probe at the hook).
- A peer appearing between request and boot lands in `terminal`; the node serves the operations API in safe mode; recovery entirely via `supersede` to the pre-restore id, then a writable restart.
- Unsupervised (no orchestrator) startup after a failed attempt stays up with the database blocked.
- Rollback content asserted equal to the last acknowledged pre-restart write.
- `retain_previous=false` recorded in the job; `cancel` after destructive work leaves the marker.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.