borgbackup / borgbackup/borg

borg2: Consider using blake3 as storage key for packs and indexes

Open
#9,943 9 comments 0 reactions 1 assignee Claimed by @ThomasWaldmann View on GitHub
Dominant language
Python
Stars
13.7k
Forks
875
Avg merge
11h 15m
Merged PRs (30d)
192

Description

This is a follow-up to the discussion in #9935.

As discussed there, Borg 2 currently uses `sha256(content)` as the storage key (object name) for packs and indexes. The rationale makes sense: simpler code, a stable algorithm, broad ecosystem support, verification without Borg (just check the `sha256sum` against the file name), and content-addressable storage.

However, while SHA hardware acceleration is widely available today, there is still a significant amount of hardware in active use that does not support SHA extensions (see [docs](https://github.com/borgbackup/borg/blob/2bf0ff118bd7bec1a7e9cc2df1abb21da3ef32fb/src/borg/archiver.py#L4670-L4687); most notably, Intel only introduced SHA extensions in 2021). On such systems, `sha256` performance is expected to be significantly lower, which is why Borg 1.1 added support for `blake2b` as ID hash, and Borg 2 recently switched that to `blake3`.

I wonder whether the mandatory use of `sha256` for pack and index IDs has a measurable performance impact on systems **without SHA hardware acceleration**, and whether that impact could justify letting repositories created with `borg2 repo-create --id-hash blake3` use `blake3` for pack and index IDs instead. As discussed in #9935, this should be possible in principle, with the main downside being the additional code maintenance burden. For example, verification without Borg would still be possible using the [official `b3sum` tool](https://github.com/BLAKE3-team/BLAKE3/tree/master/b3sum).

As always, this depends on whether `sha256` is actually the limiting factor in a given workload. As Thomas pointed out in #9935, it is unlikely to matter when writing to slow remote storage such as cloud providers, where other bottlenecks dominate. However, it could become relevant on fast local storage.

I'm currently migrating a Borg 1 repository on an Intel Core i5-8265U (8th generation, no SHA extensions), where `borg transfer --from-borg1` appears noticeably slower than expected. I haven't yet isolated whether `sha256` contributes significantly to that slowdown, but I plan to run some benchmarks when I have the time and will post the results here if they provide useful data.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.