borg2 b24: benchmarking needed
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 875
- Avg merge
- 11h 15m
- Merged PRs (30d)
- 192
Description
borg2 chunkers got another round of tuning recently, but it depends a lot on the CPU of the host system.
To select the algorithm it just uses env vars now.
It should also select a hardcoded fastest algorithm per platform now (x86-64 and arm64) when no env var is set.
You can help by doing some measurements on your machine(s) - x86-64, arm64 and also others (others will run the generic code that works on any platform). Try to have a relatively idle machine.
Use borg 2.0.0b24.
These are some command lines that can be used for benchmarking, I put a comment into the line I consider fastest currently:
```
# Normal chunking - only fastcdc and buzhash64 performance is interesting here.
# X86-64 (AMD/Intel)
BORG_FASTCDC_KERNEL=scalar BORG_BUZHASH64_KERNEL=scalar borg benchmark cpu --chunking # fastest!?
BORG_FASTCDC_KERNEL=blockwise BORG_BUZHASH64_KERNEL=blockwise borg benchmark cpu --chunking
BORG_FASTCDC_KERNEL=avx2 BORG_BUZHASH64_KERNEL=avx2 borg benchmark cpu --chunking
BORG_FASTCDC_KERNEL=avx512 BORG_BUZHASH64_KERNEL=avx512 borg benchmark cpu --chunking
BORG_FASTCDC_KERNEL= BORG_BUZHASH64_KERNEL= borg benchmark cpu --chunking # correct autoselect?
# Cryptographic chunking - only the *-aes chunker performance is interesting here.
# X86-64 (AMD/Intel)
BORG_AES_CHUNKER_KERNEL=evp borg benchmark cpu --chunking
BORG_AES_CHUNKER_KERNEL=aes-ni borg benchmark cpu --chunking
BORG_AES_CHUNKER_KERNEL=vaes borg benchmark cpu --chunking # fastest!?
BORG_AES_CHUNKER_KERNEL= borg benchmark cpu --chunking # correct autoselect?
# Normal chunking - only fastcdc and buzhash64 performance is interesting here.
# ARM64 CPUs (incl. Apple Silicon)
BORG_FASTCDC_KERNEL=scalar BORG_BUZHASH64_KERNEL=scalar borg benchmark cpu --chunking
BORG_FASTCDC_KERNEL=blockwise BORG_BUZHASH64_KERNEL=blockwise borg benchmark cpu --chunking # fastest!?
BORG_FASTCDC_KERNEL=neon BORG_BUZHASH64_KERNEL=neon borg benchmark cpu --chunking
BORG_FASTCDC_KERNEL= BORG_BUZHASH64_KERNEL= borg benchmark cpu --chunking # correct autoselect?
# Cryptographic chunking - only the *-aes chunker performance is interesting here.
# ARM64 CPUs (incl. Apple Silicon)
BORG_AES_CHUNKER_KERNEL=evp borg benchmark cpu --chunking
BORG_AES_CHUNKER_KERNEL=aes-arm64 borg benchmark cpu --chunking # fastest!?
BORG_AES_CHUNKER_KERNEL= borg benchmark cpu --chunking # correct autoselect?
```
Contributor guide
Research direction
Start with borg 2.0.0b24 and run the provided `borg benchmark cpu --chunking` commands on relatively idle x86-64, arm64, and other machines. Compare scalar, blockwise, vectorized, AES, and automatic selections; done means recording measurements that identify suitable platform defaults when no environment variable is set.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100