Restore & backup verification (Restore CRD + restore runner + integrity/test-restore)

Summary

Add a first-class Restore path and backup verification. Today there is no restore (grep restore matches only generated files) and no integrity check — backups run and we hope they're good. This is also where the operator delivers value the gobackup CLI does not: gobackup is backup-only and has no general restore command, so implementing restore here is genuinely additive.

Why it can't just shell out to gobackup

gobackup produces and uploads an artifact but does not restore. Restore must: (1) download the artifact from the Storage backend (reuse existing storage creds/config), (2) decrypt + decompress (reverse encodeWith/compressWith), (3) apply it with the DB-native tool (pg_restore/psql, mysql, mongorestore, redis-cli, etcdctl snapshot restore, influx restore). That logic lives in a small restore runner image the operator schedules as a Job.

Proposed design

Restore CRD

apiVersion: gobackup.io/v1
kind: Restore
spec:
  source:
    backupRef: { name: nightly-pg }     # inherits storages + encode/compress
    select: { mode: latest }            # latest | timestamp | fileName
  target:
    databaseRef: { name: pg-restore-target }
  overwrite: false                       # safety: refuse to clobber unless explicit

Status mirrors Backup (phase / startTime / completionTime / resolved artifact + checksum / message / logs on failure), reusing the BackupRunStatus patterns in backup_controller.go.

Restore runner image

Minimal gobackup-restore image: takes the generated config (storage creds + encode/compress) + a target DSN, streams the artifact down, pipes decrypt → decompress, execs the right native client by target.type. Phase 1 ships PostgreSQL + S3 end-to-end; expand behind a published support matrix.

Controller

RestoreReconciler (sibling to BackupReconciler): validate source/target, build the runner Job (same secret-mount pattern as buildJobTemplate), watch the Job, surface status/logs. One-shot; add a finalizer so an in-flight restore isn't orphaned.

Verification (opt-in per Backup)

spec:
  verify: { mode: integrity }   # off | integrity | restore

• integrity (cheap, recommended default): after a successful backup, re-download the artifact and verify checksum + that it decrypts/decompresses. Catches silent corruption, bad keys, truncated uploads.
• restore (gold standard): periodically restore the latest artifact into an ephemeral scratch DB and assert it loads (optional validationQuery). "Tested restores." Heavier; own schedule.

Verification feeds Status.Conditions (BackupVerified) and metrics (see observability issue).

Phases

1. Restore CRD + RestoreReconciler + runner image (Postgres + S3) + status/logs.
2. verify.mode: integrity.
3. Expand restore matrix (MySQL/Mongo/Redis/etcd; GCS/Azure).
4. verify.mode: restore (scheduled scratch-DB test restore).

Risks

• Destructive op: overwrite: false default, default target to a different Database, loud event before applying, dry-run support.
• Engine sprawl: scope by a published support matrix; fail fast on unsupported type.
• New runner image to build/scan/publish — keep tiny, reuse official client binaries.

Success criteria

• A Restore CR restores a Postgres backup from S3 into a target DB, with status + logs, from one manifest.
• A corrupted/truncated artifact is caught by verify: integrity and surfaces as BackupVerified=False + event + metric — before a human notices.
• Documented, enforced support matrix.