borg2: Consider using blake3 as storage key for packs and indexes
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 875
- Avg merge
- 11h 15m
- Merged PRs (30d)
- 192
Description
This is a follow-up to the discussion in #9935.
As discussed there, Borg 2 currently uses `sha256(content)` as the storage key (object name) for packs and indexes. The rationale makes sense: simpler code, a stable algorithm, broad ecosystem support, verification without Borg (just check the `sha256sum` against the file name), and content-addressable storage.
However, while SHA hardware acceleration is widely available today, there is still a significant amount of hardware in active use that does not support SHA extensions (see [docs](https://github.com/borgbackup/borg/blob/2bf0ff118bd7bec1a7e9cc2df1abb21da3ef32fb/src/borg/archiver.py#L4670-L4687); most notably, Intel only introduced SHA extensions in 2021). On such systems, `sha256` performance is expected to be significantly lower, which is why Borg 1.1 added support for `blake2b` as ID hash, and Borg 2 recently switched that to `blake3`.
I wonder whether the mandatory use of `sha256` for pack and index IDs has a measurable performance impact on systems **without SHA hardware acceleration**, and whether that impact could justify letting repositories created with `borg2 repo-create --id-hash blake3` use `blake3` for pack and index IDs instead. As discussed in #9935, this should be possible in principle, with the main downside being the additional code maintenance burden. For example, verification without Borg would still be possible using the [official `b3sum` tool](https://github.com/BLAKE3-team/BLAKE3/tree/master/b3sum).
As always, this depends on whether `sha256` is actually the limiting factor in a given workload. As Thomas pointed out in #9935, it is unlikely to matter when writing to slow remote storage such as cloud providers, where other bottlenecks dominate. However, it could become relevant on fast local storage.
I'm currently migrating a Borg 1 repository on an Intel Core i5-8265U (8th generation, no SHA extensions), where `borg transfer --from-borg1` appears noticeably slower than expected. I haven't yet isolated whether `sha256` contributes significantly to that slowdown, but I plan to run some benchmarks when I have the time and will post the results here if they provide useful data.
Contributor guide
Assessment
This issue has not been assessed yet.