HarperFast / HarperFast/harper

[Epic] Restore a get_backup archive into a running Harper (incl. Fabric)

Open
#2,632 0 comments 0 reactions 1 assignee Claimed by @cb1kenobi View on GitHub
area:operations-api area:storage enhancement
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

Design note: `docs/proposals/archive-restore.md` on branch `design/archive-restore` (four planning-review rounds, the last two by codex `gpt-6-astra` at extra-high effort; both returned `chosen-approach-sound`). This epic tracks the work the note breaks down in its §9. It supersedes the rationale on #995 (closed 2026-07-31, "we don't want to introduce restoring backups while Harper is running"); #1831 landed `restore_backup` doing exactly that the next day.

## The problem

`get_backup` streams a full point-in-time snapshot of a database as `.tar`/`.tar.gz` (`createBackupStream`, `dataLayer/rocksdbBackup.ts:689`). Nothing consumes it: the archive's embedded README tells the operator to stop Harper and hand-copy files into the database directory and each blob root. On Fabric that is unusable three ways — the operator has no filesystem access, `target_database` is refused while Harper runs (`:500`, `:537`), and "stop the server" is a host-manager action. The third point is not specific to archives: any database a component declares tables in is component-held, `verifyDatabaseClosed` (`:624`) returns 409 for it, and the 409 points at a CLI the Fabric operator cannot run. So `restore_backup` itself cannot restore the databases people actually use on Fabric today.

## The approach

A `get_backup` archive and a managed backup differ only in format, and the conversion is two existing calls: an extracted stream backup "unpacks into a directory that opens directly as a RocksDB database" (rocksdb-js `docs/backups.md`), and `db.backup(repo)` on an opened directory produces a managed backup — which is exactly what `createBackupOffline` already does (`dataLayer/rocksdbBackup.ts:959-990`). Everything else an archive restore needs is already the managed restore path: lock + marker, `verifyDatabaseClosed`, `backups.restore(purgeAllFiles)`, index-aware blob restore, `verify_backup`, and the offline CLI. What that path lacks — a way to run for a held database on Fabric — it lacks for managed backups too.

So:

1. **`import_backup`** converts an archive (multipart upload or a pull from S3/HTTPS) into a verified managed backup id. No restart.
2. **A shared publication protocol** under which a restored database becomes visible: the restore lock as a real name reservation, private open, expected-table and column-family inventory checks, a restore-strict elected transaction-log replay, a blob-completeness decision, durability barriers, then publication. It carries **restore into a new name online** (create-if-absent) and upgrades in-place `restore_backup` for unheld databases.
3. **A restart route on `restore_backup`** (`on_restart: true`) for component-held databases: a durable intent state machine, a boot hook that runs the restore before components load, a pre-restore backup captured once as the rollback source, no-exit retries (self-managed installs have no supervisor), a terminal state that boots in safe mode, and `resume`/`supersede`/`cancel` entirely through the API.

The invariant every piece serves (note §3): an acknowledged restore publishes exactly one complete, recoverable generation — engine data, schema, replayed transactions, index-addressed blobs — and no reader or creator can observe an intermediate one; the recovery source stays available until completion is durable; a restore under a new name holds an exclusive reservation from the first byte; a retry preserves the original rollback identity and never overwrites a published generation.

Restore is refused on any node with peers (`hdb_nodes` rows other than self, or `replication.routes` entries), re-checked at the boot hook and fenced during online restores. The `system` database, LMDB archives, and v4 migration are out of scope.

## Work items

| # | Issue | Depends on |
|---|---|---|
| 1 | #2633 — capability manifest as the archive's first entry, and in managed backup manifests | — |
| 2 | #2634 — `beginRestore()` truncates the marker; write temp → fsync → rename | — |
| 3 | #2635 — boot-time job reconciliation with process-instance ownership | — |
| 4 | #2636 — repository operations without a loaded database, per-repository management lock, restore pins | — |
| 5 | #2637 — publication protocol and online create-if-absent (`target_database` online) | 2, 4, 8, 9 |
| 6 | #2638 — `on_restart` route for component-held databases | 3, 4, 5, 8, 9 |
| 7 | #2639 — `import_backup` | 1, 4, 9 |
| 8 | #2451 — database generation on restore (or minted by 5 and 6 at the same point) | — |
| 9 | #2640 — refuse an engine-only restore over live blob roots without an explicit opt-in | — |

Items 1–4 and 9 each fix something on `main` today. The two PRs that carry the user-visible feature are item 5 (create-if-absent) and item 6 (replace an existing, held database). #2031 is a dependency only in the minimum form item 4 carries; #2033 is adjacent, not a dependency.

## Rollout

Producer manifest → repository operations and pins → publication protocol with restore into a new name (stands alone, and is the safest mode) → restart route → feature-gated `import_backup` → Fabric canary evidence, including GTM behavior across the drained restart.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.