HarperFast / HarperFast/harper

import_backup: convert a get_backup archive (upload or pull) into a managed backup

Open
#2,639 0 comments 0 reactions 1 assignee Claimed by @cb1kenobi View on GitHub
area:cli area:operations-api area:storage enhancement
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

Part of #2632 (item 7). Design note §5.2–§5.5 and §8 in `docs/proposals/archive-restore.md` on branch `design/archive-restore`. Depends on #2633, #2636, #2640. This is the operation #995 asked for; it is filed fresh rather than reopening #995 because the design has changed shape since.

## Problem

Nothing consumes a `get_backup` archive. The only route is the README's "stop Harper and hand-copy files into the database directory and each blob root", which requires reproducing the blob root-index mapping by hand (records persist the root index, not the path) — a silent-mis-addressing failure mode — and is impossible on Fabric.

## What to build

`import_backup` converts an archive into a **verified managed backup id** in the database's repository. No restore, no restart. Two sources, one pipeline after the bytes are on disk:

- **Push**: multipart streaming upload (the `deploy_component` machinery; `server/serverHelpers/multipartParser.ts`). The declared uncompressed size is a field part and is **advisory** (early reject only — it is client-supplied). The HTTP worker streams to staging; when the upload is complete the **main thread adopts** the staging area (owner record rewritten to the import id, fsynced) *before* the job row is inserted, so a worker that dies between enqueue and a post-enqueue rewrite cannot have its staging swept from under the job.
- **Pull**: `url` (https) or the `s3` object `import_from_s3` takes. A standard job (scalar request); resumable with `Range`; never crosses the Fabric ingress. Primary Fabric form.

**Staging** is one area beside the repository (`//.import//`) so the blob snapshot can hard-link; entry filtering is explicit (reject `..`, absolute paths, symlinks, hardlinks, device nodes) and bounded by entry count, metered uncompressed bytes, per-entry size, path length, duplicate names — the multipart parser deliberately enforces no size cap (`multipartParser.ts:15-17`), so inode exhaustion is otherwise unbounded. Failure removes staging eagerly; the main thread sweeps worker-owned staging on worker exit; a boot sweep removes staging no live job owns.

**Disk gate**: one **ledger** across staging, conversion and any later restore, grouped by the quota or filesystem each path lands on, debited by every byte written, re-based only on a sample newer than the last debit. `getStorageSpaceStats()` under `basis: 'quota'` returns the same host-manager snapshot for ~90 s (`server/storageReclamation.ts:19-22`), so re-reading it is not a gate; `EDQUOT`/`ENOSPC` is the authoritative stop. Blob bytes count unless the hard-link actually succeeded (`EMLINK`/`EPERM` fall back to copy, `dataLayer/blobBackup.ts:59-80`).

**Validation and conversion**, in a purpose-built child process (engine level only, no resource layer; bounded by wall-clock, output size, heap, block cache, file descriptors, and `oom_score_adj` on Linux):

1. `RocksDatabase.open` on the staged directory.
2. `validateTransactionLogStore(/transaction_logs, { strict })` — the check `backups.verify` runs.
3. Walk `__dbis__` for the expected table set and read the physical column-family inventory from `OPTIONS-*`; a catalogue row whose family is missing fails here.
4. `db.backup(/, { transactionLogs: true })` — the conversion (`createBackupOffline` already does this against a closed directory, `dataLayer/rocksdbBackup.ts:959-990`).
5. Opt-in: checksum walk; blob-reference scan.

Then, back in Harper: `snapshotBlobs` from the staged `blobs//` trees, the manifest (`blobs`, expected table set, inventory, archive digest, producer fields from #2633, `imported_from`), directory-only `verify_backup` (#2636), remove staging. Signal death, timeout, malformed output and non-zero exit are all failures; the serving process survives each. The strict Harper replay is **not** here (a replay writes and needs the resource layer); it belongs to the publication protocol in the restore routes.

**Pull-source security** (see the note's §5.2 for the full list):

- Never ambient credentials: the S3 client is built with explicit credentials only (as `utility/AWS/AWSConnector.js` does today); the `url` client carries no Harper TLS client certs and no inherited headers; `GET`, `https` only; other schemes rejected.
- No secrets in the job row: inline long-lived keys are **rejected**; accept a secrets-store reference (as `deploy_component` resolves registry/git credentials, `components/secretOperations.ts:337`) or a presigned URL, redacted to origin and path in every log, error and result.
- One fetch helper shared with `csv_url_load` and `deploy_component package=`, so the SSRF policy tracked for `csv_url_load` applies here too; the default allow/deny list must keep today's private-endpoint pulls working.
- Optional `sha256` pin; digest and `imported_from` always recorded.
- Idle and total download timeouts; one import per database and per instance, with the lock owned by the **main thread** (a process-owned flock taken on an HTTP worker strands on the routine worker restarts `deploy_component`/`restart_service` perform — `dataLayer/restoreMarker.ts:41-46`).

Engine-only archives (`exclude_blobs`) import fine; their manifest records `blobs: false` and the single rule in #2640 governs restoring them in place.

## Acceptance

- CLI `get_backup` → `import_backup` (push, and pull from S3 and https) → `verify_backup` on the imported id, including into a database that does not exist yet.
- Malformed, truncated, legacy and pre-manifest archives; inode exhaustion; a gzip bomb; a stale quota sample with an undeclared upload (ledger holds); `EDQUOT` mid-write cleans up.
- A worker killed between enqueue and adoption (staging survives for the job) and between adoption and enqueue (staging swept); an HTTP worker restart mid-upload releases the lock and sweeps staging.
- Inline S3 keys rejected; a presigned URL never appears un-redacted in logs, errors or results; the S3 client never falls back to the default provider chain (test with ambient env credentials present).
- Unrelated request latency measured during extraction and conversion.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.