HarperFast / HarperFast/harper

get_backup archives carry no machine-readable manifest; add a capability manifest as the first tar entry (and to managed backup manifests)

Open
#2,633 0 comments 0 reactions 1 assignee Claimed by @cb1kenobi View on GitHub
area:operations-api area:storage enhancement
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

Part of #2632 (item 1). Design note §7.1 in `docs/proposals/archive-restore.md` on branch `design/archive-restore`.

## Problem

A `get_backup` archive contains only human-readable READMEs (`streamedBackupReadme`, `dataLayer/rocksdbBackup.ts:928`; `blobsReadmeContent`, `dataLayer/blobBackup.ts:228`). Nothing machine-readable identifies the producing Harper version or the storage format, so a future importer cannot refuse an incompatible archive it cannot identify. Every archive produced today is unidentifiable, and those are the archives people will try to restore — the set only grows, which is why this ships first.

The managed backup manifest has the same gap: `dataLayer/backupManifest.ts` records only `{ backupId, blobs, completedAt }`.

## What to build

- A manifest as the **first** tar entry of every `get_backup` archive. It must be first: reaching the end of a `.tar.gz` requires inflating the whole stream, while the first entry is readable after a few KB.
- Fields: archive-schema version; producing Harper version (for logging); rocksdb-js major; transaction-log format version; blob-format features present (compression, marker types — deflate-compressed blob bodies from #2443 are unreadable by a Harper without that support and no engine check catches them); source database name; whether blobs are included; blob root count; the **names** of roles holding grants on the database (names only — the archive carries no role catalogue, and the restore side reports which names are absent locally rather than creating anything).
- **Compatibility rule: capability flags, not a scalar version.** The reader refuses when the target lacks any capability the archive declares. (An earlier draft cited #2046 as "the 5.2.0 upgrade is one-way"; that issue's resolution says the migration is additive and the binary was blocked on a hidden downgrade prompt, and it concerns `system.hdb_info`, not user-database format. The real hazards are RocksDB `format_version`/OPTIONS, structon struct-mode records (`DESIGN.md:17-25`), the transaction-log format version, and blob file format.)
- The same fields on the managed backup manifest. A managed backup whose manifest predates them is accepted (same-instance lineage) and reported as `format: unidentified`.
- **Pre-manifest archives**: accepted only under an explicit super-user override that records the operator's assertion of provenance; the result reports the archive as unidentified. Refusing them outright would make the importer useless for exactly the backups already on operators' disks.

Implementation note: this unifies `createBackupStream`'s two branches. The `excludeBlobs` path hands the whole archive to the binding, which gzips it directly (`dataLayer/rocksdbBackup.ts:704-709`), so there is no tar-stream pack to prepend an entry to; both paths route through the plain-tar-then-gzip assembly `streamBackupWithBlobs` already uses.

## Acceptance

- Every `get_backup` archive (with and without `exclude_blobs`) starts with the manifest entry; `tar tf` shows it first.
- Unit: capability-flag refusal matrix, including a pre-manifest archive with and without the override, and a managed manifest without the new fields.
- `create_backup` manifests carry the producer fields; `list_backups` surfaces them.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.