HarperFast / HarperFast/harper

restore_backup: shared publication protocol and online restore into a new database (create-if-absent)

Open
#2,637 0 comments 0 reactions 1 assignee Claimed by @cb1kenobi View on GitHub
area:operations-api area:storage enhancement
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

Part of #2632 (item 5) — the **create-if-absent** PR. Design note §6 and §5.7 in `docs/proposals/archive-restore.md` on branch `design/archive-restore`. Depends on #2634, #2636, #2640, and either #2451 or minting a generation here.

## Problem

`target_database` is refused while Harper runs (`dataLayer/rocksdbBackup.ts:500`, `:537`) by policy, not because the runtime cannot publish a new name — `create_database` → `database()` → `signalSchemaChange` does that today (`dataLayer/harperBridge/ResourceBridge.ts` `createSchema`, `resources/databases.ts:1897-1935`). Restore-into-a-copy is the safest disaster-recovery move (inspect the copy, then swap or copy across), and on Fabric it is currently unreachable.

Simply lifting the refusal is not enough, because today's publication is weak in three ways:

- The restore marker is a **check, not an exclusion**. `database()` calls `throwIfBlockedByRestore` and then opens without holding the lock (`resources/databases.ts:1932-1937`), so an opener that passes the check, pauses, and resumes after the marker lands opens the tree being restored; the startup scan snapshots its blocked set once and then opens each directory (`:577-608`); and `restoreBackupOffline` checks the target is absent *before* it reserves (`dataLayer/rocksdbBackup.ts:1008-1020`), so a concurrent `create_database` can lose the database it just made.
- The current online `restoreBackup` clears its marker and lets the ordinary reload publish the directory (`:605-611`) — no private open, no awaited replay, no check that the tables the backup should contain actually opened. "No exception thrown" is the #2095 false-green shape.
- `initStores` recreates a missing column family as an empty store when the catalogue still names it (`resources/databases.ts:1091-1096`), so "opened successfully" cannot prove a family existed.

## What to build: one publication protocol

Used by online restore into a new name, by in-place online restore of an unheld database, and (item 6) by the restart route.

1. **Reservation first.** `beginRestore(targetDir)` before the first byte lands; the target is re-checked for absence (or, in place, for closure) *inside* the reservation. On-demand opens and the scan's per-directory open take the per-database restore lock in **shared** mode (`tryFileLock(file, true)`) across check-and-open, releasing once the handle is registered; restore holds it exclusive. `dropDatabase` keeps serializing on the same lock. All of this is on cold open paths — no per-request work.
2. **Apply without finishing.** Split `restoreBackupOffline` into an apply helper (lock, marker, purge, engine restore, blob restore) that does **not** call `completeRestore`, so the marker survives a failed validation; the CLI keeps its current shape by calling apply and then completing.
3. **Inventory, then private open.** Read the physical column-family inventory from the restored directory's `OPTIONS-*` file and check every family the catalogue requires is present, *before* any create-capable open. Then `initStores(path, root, db, { destination, openedStores })` and check the expected table set is a subset of what enumerated and every store opened.
4. **Restore-strict elected replay**, in the job worker (never on a serving thread — `replayLogs` runs synchronously behind its Promise, `resources/replayLogs.ts:60-136`, and a `threads: 0` install serves from main). `await replayLogs(root, destination, { elected: true, policy: 'restore' })`: elected mode already rejects a failing tail but still tolerates undecodable entries (`skipped++`, `:183-201`), so the restore policy rejects when `skipped > 0` or `discardedWrites > 0`, and gates `purgeAgedLogs` off (it otherwise deletes a restored log older than `logging.auditRetention` before replay, `:83`).
5. **Blob completeness.** A blob reclaimed before its directory was enumerated is absent from the source with no marker (`dataLayer/blobBackup.ts:30-40`). Replacement restores run the O(data) blob-reference scan unless the request carried `accept_unverified_blobs: true`; the result records `blob_integrity: verified | unverified | failed`. `target_database` reports it and does not require it.
6. **Peer fence and re-check.** Core exposes "a restore reservation is held for X"; `add_node`/`set_node`/`add_node_back` and routes-config changes refuse while it is; peers are re-checked after replay and before `completeRestore`.
7. **Durability barriers.** Close private handles and surface close errors; fsync restored blob files and their directories; fsync manifests. Today's copies do not sync (`dataLayer/blobBackup.ts:59-80`, `:192-211`; `dataLayer/backupManifest.ts:47-57`). State the Windows directory-fsync limit rather than hide it.
8. **Finalize**: `completeRestore`, then the rescan that publishes. Mint or consume the generation (#2451) between steps 2 and 8.

Then lift the `target_database` refusal when the target directory is absent; the offline form's "never purge an existing database of that name" (`:1008-1013`) applies unchanged. Engine-only backups follow the single rule in #2640.

## Acceptance

- `restore_backup database=X target_database=Y` online, with blobs, records and blobs readable afterwards; roles are not created (a new name has no grants — expected).
- An opener paused between `throwIfBlockedByRestore` and its open while a restore reserves the name is held off; a rescan and a `create_table` racing an online restore into a new name lose to the reservation, not the other way round.
- A backup whose catalogue names a dropped column family fails validation; a log with valid framing and an undecodable value is rejected before publication; a blob missing without a marker is caught by the scan and reported under the waiver.
- A peer or route added after online admission is refused while the name is reserved, and caught by the pre-publication re-check if it slips through.
- Unrelated request latency measured during publication and replay, including on a `threads: 0` install.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.