borg2 check: implement repository repair (index and pack rebuild)
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 875
- Avg merge
- 11h 15m
- Merged PRs (30d)
- 192
Description
# borg2 check: implement repository repair (index and pack rebuild)
Refs #9696, #8466, #8476, #9901, #9825. Tracking: #9998.
## Problem
The read-only repository check is done: it hashes `index/` and `packs/` objects via `Store.hash`,
checks the index first and the packs only if the index is intact, and never rebuilds the index on
that path. But when it finds damage, there is no repair. `Repository.check(repair=True)` just logs
and returns:
```python
# repository.py, check()
elif repair:
logger.error("Finished ... repository check, errors found (repository repair not implemented).")
```
and the corrupt-index branch has the placeholder for the missing part:
```python
# TODO: --repair will rebuild the index from the packs here instead of stopping (refs #8572).
logger.error("Repository index is corrupted and must be repaired; skipping the pack check.")
```
A repo with a corrupt `index/` fragment or a corrupt pack therefore cannot be repaired at the
repository level. This ticket splits that work out of #9696 ("... to be continued").
## Scope
Repository-level repair only: rebuilding the chunks index and salvaging packs. Archives repair
(dropping archives with corrupt/lost metadata, recreating lost entries) already exists, and missing
file-content chunks heal on the next `borg create`. Both are out of scope.
## Work items, in order
1. **Command surface.** Decide before writing repair code: keep it under `borg check --repair`, or
split off a read-only `borg check` and a separate `borg repair` with explicit targets
(`--index`, `--packs PACKID...`) (#9696). The items below assume the existing `check` paths; only
the CLI shape depends on this.
2. **Index repair from headers.** Rebuild the chunks index by range-reading pack object headers and
skipping the payloads, recording `chunk id -> (pack id, offset, size)`. Reading full packs is too
slow to be the routine repair path (#8476, #8572). Replaces the TODO in the corrupt-index branch.
3. **Do not trust the headers blindly (#9901).** Headers are unauthenticated, so a tampered header
(wrong id or sizes) would poison the rebuilt index. Authenticate on first real read (AEAD binds
the id to the content) and/or verify ids for mismatching entries. Part of item 2, not a follow-up.
4. **Pack repair.** For a pack that fails `Store.hash`, keep every object that still
AEAD-authenticates in a new pack, drop the corrupt pack, and update the index. Objects that no
longer decrypt are already gone; drop them from the index so the affected archives report missing
chunks and heal on the next create (#8476, #8572). Needs item 2.
5. **Consume the persisted corrupt-pack list.** A read-only check records corrupt pack ids in
`cache/checked-packs` (#9897; PR #9925 makes them survive a completed check). Repair reads that
list and touches only the flagged packs, then clears their records once fixed (#9696). Needs
PR #9925 plus items 2 and 4.
6. **Keep the rebuilt index (#8466).** `ArchiveChecker.finish()` deletes the cached chunks index
after repair, so the next command pays for a slow rebuild. Write the index rebuilt in item 2 back
instead. Needs item 2.
7. **Repair progress (#9443, #8009).** The read-only check shows per-loop percentages; the repair
paths (header scan, pack salvage, verify-data delete pass) need the same. Needs items 2 and 4.
## Out of scope (tracked elsewhere)
- Vanished-pack detection, cheap cloud check mode (#9898, PR #9925).
- Reporting/UX: findings+repairs summary (#2397, PR #9891), grouped missing-chunk reports (#9218,
PR #9965), Ctrl-C at safe boundaries (#7893, PR #9966).
- Post-corruption docs and acknowledging known-lost chunks so check/compact stop returning exit 2
(#9825).
- Repair from a redundant related repo (#6584, milestone 2.1).
## Data-safety constraints
- Never delete an object after a single failed read; a second read must also fail first. Transient
network or RAM errors must not destroy data.
- A rebuilt index must never claim objects that do not exist. A lying index breaks `borg create`
dedup (#8476).
- An interrupted repair must leave the repository consistent (#7893).
Contributor guide
Research direction
Start with repository.py check(), Store.hash, and the corrupt-index branch to understand the existing read-only paths. Then trace ArchiveChecker.finish() and the persisted cache/checked-packs records before deciding the command surface. Done means safe header-based index rebuilding, authenticated pack salvage, persisted-list handling, index retention, progress reporting, and consistency after interruption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100