borg2 check: overview / tracking ticket

This ticket collects all `cmd: check` tickets (especially the ones in the 2.0.0b23 milestone)
into one place: a summary of how borg2 `check` is supposed to work, plus a checkbox list of the
concrete work items. Ticked items are already in master (commit linked); unticked items are
still to do (open PR linked where one exists).

Tracked tickets: #9696 #9825 #9898 #9901 #9218 #9443 #8009 #7893 #2397 #8466 #8476 (b23),
#8478 (b24), #6584 (2.1), plus the already-closed #9897 #9411 #8518 #8517 #8559 #8297 #1931 #7062.

## How borg2 check is supposed to work

### What changed vs. borg 1.x

borg 1.x check had a repository part (crc32-check all segment entries, always rebuild the
repository index, run server-side for ssh repos) and an archives part (read all archives, check
chunk presence, optionally `--verify-data`). Much of that design no longer applies:

- A borg2 repository is a borgstore store. There is no borg process on the server side
(except for `rest:` served by `borg serve --rest`), so "run the check on the server" is
replaced by something better: **content-addressed verification**.
- Pack files (`packs/`) and chunk index fragments (`index/`) are named by the
sha256 of their content ([`fd177535`](https://github.com/borgbackup/borg/commit/fd1775356170552b48107efcab569e8a49197271)).
So "is this object intact?" is simply `Store.hash(name) == name`. The REST backend computes
the hash server-side, so a full repository check of a remote repo transfers no pack data.
- The chunks index (chunk id → pack id, offset, size) lives in the repository and is the
authoritative record of which chunks exist and where.
- There is no `chunks_healthy` anymore (#8559): an item's `chunks` list always holds the
correct chunk ids, even when chunks are missing. Missing chunks are handled dynamically at
read time (zeros or error), and if a later `borg create` re-produces a missing chunk, all
old archives referencing it are healed instantly — no repeated `check --repair` runs.

### Phase 1: repository check (cheap, read-only, no key needed)

1. Verify all `index/` fragments via `Store.hash` (small, always checked in full).
2. If (and only if) the index is intact, verify all `packs/` the same way.
3. Partial checks (`--max-duration`) resume from a persisted set of checked packs
(`cache/checked-packs`, #9897), with periodic checkpoints. Packs recorded as corrupt are
always re-verified, never skipped.
4. The read-only check **never rebuilds the index**: reading every pack is far too slow and
expensive for a routine check. A corrupt index is reported and left for repair.

This phase is designed to be a routine, cron-able job: no key, no prompt, bounded time,
meaningful exit code. Its findings (corrupt pack ids) should be persisted so a later repair
can consume them instead of re-scanning everything (PR #9925).

### Phase 2: archives check (client-side, needs the key)

1. Ask for the passphrase up front, not hours into the run (#1931).
2. Check the manifest (rebuild if missing/corrupt), the archives directory, each selected
archive's metadata object and item metadata stream (robust resync on damage), and each
item's file-content chunk presence against the (verified) chunks index.
3. `--verify-data`: additionally read, decrypt (AEAD-authenticate) and decompress every
object — full cryptographic verification, detects malicious tampering too (for encrypted
repos, against attackers without the key).
4. `--find-lost-archives`: full scan of all objects for archive metadata without an archives
directory entry; with `--repair`, recreate the entry (possible until `borg compact` removes
the data).

A normal (non-repair) archives check trusts the hash-verified index and does not rebuild it
(#8476); only repair rebuilds from the actual packs.

### Repair philosophy

- **Only repair what is known to be broken.** A repair should be able to consume the corrupt
pack list a previous cheap check produced, instead of re-scanning all packs (#9696).
- **Index repair** rebuilds the index by scanning pack object headers only (seek over
payloads, no full read) — but must not blindly trust the unauthenticated headers (#9901).
- **Pack repair** salvages objects out of a corrupt pack: every object that still decrypts
(AEAD-authenticates) goes into a new pack, the corrupt pack is dropped, the index updated
(#8476, #9696, refs #8572).
- Defect chunks are only deleted after a second read attempt also fails (transient errors —
network, RAM — must not destroy data).
- Command split is open for discussion: `borg check` strictly read-only, all
repairing/rebuilding in `borg repair` (with e.g. `--index`, `--packs PACKID...`) (#9696).

### Design goals

- **Data safety**: never delete data on a single failed read; never let a rebuilt index claim
objects that do not exist (fatal for `borg create` dedup, #8476); verify before trusting
(#9901); a interrupted check/repair must leave a consistent state (Ctrl-C at safe
boundaries, #7893).
- **Usability for admins**: routine checks need no key and no prompts; progress indication
everywhere (#9443, #8009); a summary of findings and repairs at the end (#2397); reports
organized around the question the admin actually has — "which files/archives are affected?"
(#9218); a documented "what do I do now?" workflow after corruption (#9825); customizable
output (#9411).
- **Performance**: server-side hashing (no transfer); no unnecessary index rebuilds (#8476);
partial/time-boxed checks; reuse of still-fresh check results (`--max-age`, PR #9925);
headers-only scans where possible; combine full-repo passes instead of reading everything
twice (#8478, ex-#8518).

## Work items

### Repository check (read-only)

- [x] Content-addressed read-only check: verify `index/` and `packs/` objects via
`Store.hash`, server-side hashing for `rest:` repos; index first, packs only if the
index is intact; no index rebuild in the read-only path (#9696, #8476, #8518) —
[`27d693c2`](https://github.com/borgbackup/borg/commit/27d693c222bbefca73aac10864b37c483dd56dd1) (PR #9794, with PRs #9783/#9789 for sha256 pack naming)
- [x] Progress indicators (percent) for the index and pack verification loops (#9443, #8009) —
[`27d693c2`](https://github.com/borgbackup/borg/commit/27d693c222bbefca73aac10864b37c483dd56dd1) (PR #9794)
- [x] Partial checks: persist a *set* of checked packs instead of a "last pack" marker, so
packs added mid-cycle cannot be skipped (#9897) —
[`bc8b3c07`](https://github.com/borgbackup/borg/commit/bc8b3c07721b348b684f76812641a0f933f4377e)
- [x] Re-verify packs recorded as corrupt instead of skipping them on resume (#9897) —
[`9543c141`](https://github.com/borgbackup/borg/commit/9543c1411c945d50bb649cf2ad6fcb5bb24b58d2)
- [x] Checkpoint the checked-packs set periodically (30 min) during long runs (#9897) —
[`a1b88e2d`](https://github.com/borgbackup/borg/commit/a1b88e2dc7f371ac879b192831d88b2bf9c36041)
- [ ] Keep pack check results after a completed check and add `--max-age` to skip packs with
a fresh intact result — makes cheap periodic checks feed later repairs and avoids
re-hashing everything on slow/cloud storage (#9696, #9898) — PR #9925 open
- [ ] Detect vanished packs: cross-check that every pack referenced from the chunks index is
present in `packs/` and has the expected size (#9898). Currently a completely missing
pack is only noticed at read time
([`cbc94ae8`](https://github.com/borgbackup/borg/commit/cbc94ae8378e9cbfc03e3ba6bc2a92ca54932faa) `PackNotFound`), by `--verify-data`, or by the `--repair` index rebuild.
- [ ] Cheap check mode for cloud repos: make full pack-sha256 verification optional; default
(or an option) could be listing packs and checking presence + size against the index,
avoiding the full data transfer (#9898)

### Archives check

- [x] Ask for the key passphrase early, before the (potentially very long) repository phase
(#1931) — [`2340e605`](https://github.com/borgbackup/borg/commit/2340e60504d8e0f85389b42fd307c52acf19be00)
- [x] Archive selection filters `--first/--last/-a/--sort-by` (#1663) and
`--older/--newer/--oldest/--newest` (#7062) —
[`b2654bc1`](https://github.com/borgbackup/borg/commit/b2654bc17de472162ad9892eb4b0668232789424)
- [x] `--format` support for the "Analyzing archive ..." output, validated up front,
`BORG_CHECK_FORMAT` env var (#9411) —
[`d7ffe748`](https://github.com/borgbackup/borg/commit/d7ffe74801506586652a2d348fd7fa03fef38269)
- [x] Non-repair archives check trusts the hash-verified in-repo chunks index — no slow index
rebuild for a routine check (#8476, #8466) —
[`27d693c2`](https://github.com/borgbackup/borg/commit/27d693c222bbefca73aac10864b37c483dd56dd1)
(earlier step: reuse the repo-part ChunkIndex, [`7288f4f9`](https://github.com/borgbackup/borg/commit/7288f4f94d99356f9f68001c638b83474938ad44), #8468)
- [x] Remove `chunks_healthy`; self-healing archives, defined read-time behaviour for missing
chunks, no all-zero patch chunks (#8559) —
[`6357f2eb`](https://github.com/borgbackup/borg/commit/6357f2ebc143281357e7ef7598498a0e1900b6b6)
- [x] `--find-lost-archives` (né `--undelete-archives`): scan for archive metadata lacking an
archives directory entry, recreate entries with `--repair` —
[`682aedba`](https://github.com/borgbackup/borg/commit/682aedba5030e90b8a78488c89dceda7d7a0e91b),
renamed in [`a48a8d2b`](https://github.com/borgbackup/borg/commit/a48a8d2bea07e3c41b650f6286a83b2d035fe63e)
- [ ] Speed up `--verify-data` combined with `--find-lost-archives`: remember archive-metadata
object ids during the verify pass and reuse them instead of a second full repo scan
(#8478, ex-#8517 — milestone b24)
- [ ] `--verify-data` reads chunks in index (hash) order, i.e. randomly across packs; read
pack-by-pack instead so each pack is fetched once, sequentially ("read data only once",
idea from #8518, folded into #9696)

### Repair

- [ ] Repository-level repair is not implemented yet — `Repository.check(repair=True)`
currently just logs "repository repair not implemented" (#9696 "to be continued",
#9825). This is the main open block of work:
- [ ] Index repair: when `index/` objects are corrupt (or the index provably lies), rebuild
the index by scanning pack object *headers* only (range reads, skip payloads), not by
reading full packs (#9696, #8476; code TODO at `repository.py` check(), refs #8572)
- [ ] Pack repair (`--repair-pack PACKID...` or similar): salvage all AEAD-valid objects of
a corrupt pack into a new pack, drop the corrupt pack, update the index — without a
full scan of all other packs (#8476, #9696)
- [ ] Let repair consume the persisted corrupt-pack list from previous read-only checks, so
an admin's `--repair` doesn't redo the cron job's scanning work (#9696; groundwork in
PR #9925)
- [ ] Decide the command split: keep everything under `borg check --repair` vs. a separate
read-only `borg check` + `borg repair --index/--packs/...` (#9696)
- [ ] Index rebuild must not blindly trust unauthenticated pack headers: a tampered header
(wrong chunk id / sizes) currently goes straight into the rebuilt index (#9901).
Mitigation ideas: authenticate on first real read (AEAD binds the id), or verify object
ids cryptographically during rebuild for suspicious/mismatching entries.
- [ ] After `check --repair`, write the freshly built chunks index back to the repository
instead of deleting it and forcing the next command into a slow rebuild (#8466 — the
read-only path no longer rebuilds at all, but `ArchiveChecker.finish()` still deletes
the index after repair)
- [ ] Repair progress indication: the `--repair` paths (verify_data delete pass, archive
rebuild, future index/pack repair) need progress output like the read-only check has
(#9443, #8009)
- [ ] Repair using redundant related repos: fetch chunks that are missing/corrupt here from a
related repo (same key material / same chunk ids), e.g.
`borg check --repair --other-repo=...`; borgstore cache vs. primary self-healing is a
variant of the same idea (#6584 — milestone 2.1)

### Reporting & UX

- [ ] Summarize findings at the end of check, and repairs actually done with `--repair`
("N problems found, M repaired") (#2397, dup #8192) — PR #9891 open
- [x] Report missing chunks grouped as chunk → files → archives instead of one log line per
(archive, file, chunk); avoids the hardlink-caused flood of repeated lines and answers
"which files are affected?" directly (#9218) — PR #9965 open
- [ ] Ctrl-C handling: first Ctrl-C stops soon but cleanly at a safe boundary (saving
checkpoint state, running `finish()`), second Ctrl-C aborts immediately (#7893) —
PR #9966 open
- [ ] Document the post-corruption workflow (#9825): what `--repair` does and does not do,
that re-running `borg create` heals reappearing chunks automatically (no second
`check --repair` needed), how to get rid of permanently lost data
(`borg recreate --exclude ...` the affected files), "considerations to fix your repo
corruption" guide
- [ ] A way to acknowledge known-lost chunks so subsequent check/compact runs stop returning
exit code 2 / error mails for damage the admin has already accepted (#9825)

### Done / resolved by the borg2 design (no action needed)

- [x] No more `borg check` requirement before prune/compact, so the `--repair` YES prompt no
longer haunts cron jobs (#8297; `BORG_CHECK_I_KNOW_WHAT_I_AM_DOING=YES` still skips the
prompt for actual repair runs)
- [x] Segment-era tickets are obsolete: there are no segments, transactions or commits in a
borgstore repo — #2454 (check with no commit), #8535/#8580 (segment refcount / index
object count mismatch), #8070 (check a single segment; nearest borg2 equivalent is the
planned per-pack repair above)
- [x] "check --repair commit prompt" (#2398) — obsolete: borg2 has no repo transaction to
commit; the equivalent admin need (see what check finds, then repair exactly that) is
covered by the persisted check-results + repair-consumes-them items above

## Milestone view (2.0.0b23)

| Ticket | Topic | State |
|---|---|---|
| #9696 | check redesign umbrella | read-only part done; **repair part is the open core work** |
| #8466 | check --repair improvements | open (index caching after repair) |
| #8476 | avoid unnecessary index rebuild | read-only part done; repair-index/-pack open |
| #9898 | vanished packs | open (PR #9925 related) |
| #9901 | index rebuild trusts pack headers | open |
| #9825 | dealing with repo corruption | open (docs + acknowledge mechanism) |
| #9218 | missing-chunk reporting | PR #9965 open |
| #2397 | summarize findings/repairs | PR #9891 open |
| #7893 | Ctrl-C | PR #9966 open |
| #9443 | repo check progress | done for read-only ([`27d693c2`](https://github.com/borgbackup/borg/commit/27d693c222bbefca73aac10864b37c483dd56dd1)); repair progress open |
| #8009 | progress counters | borg1 wontfix; borg2 covered by #9443 items |
| #9897 | set-of-checked-packs | done ([`bc8b3c07`](https://github.com/borgbackup/borg/commit/bc8b3c07721b348b684f76812641a0f933f4377e)) |
| #9411 | check --format | done ([`d7ffe748`](https://github.com/borgbackup/borg/commit/d7ffe74801506586652a2d348fd7fa03fef38269)) |

---

made by Claude Fable 5

This is a broad borg2 check and repair tracking ticket rather than a self-contained task. Start by selecting one unticked work item, then read its linked issue or PR; for the repair work, the payload points to the check() TODO in repository.py and PR #9925. Done means the selected checkbox item is implemented and its linked requirements are satisfied.

Written by the indexing model from the issue text.