Improve hashing and verification performance
- Dominant language
- Rust
- Stars
- 659
- Forks
- 36
- Avg merge
- 18m
- Merged PRs (30d)
- 2
Description
The speed of torrent creation and verification could use some work. I picked SHA1 and MD5 implementations basically at random, and there are lots of potential performance bottlenecks in how I hash and verify pieces.
Also, #440 reports performance problems when linking against musl on Linux. If allocation, I/O, or threading is slow with musl, then general performance improvements might cause less allocation and/or I/O (I don't do any threading) which might lessen the performance impact of linking against musl.
First thing will be to set up repeatable benchmarking, so there's stable baseline to compare against. Next there are a bunch of things that are worth trying.
- [x] Split crate into binary and library, to make benchmarking possible
- [x] Set up benchmarking
- [ ] Set up flamegraph generation
- [ ] Tests which exercise incomplete reads and uneven files, and probably some known good infohashes of large files and directories of files. (They can be generated with an algorithm, so they don't need to be saved in the repo.)
- [ ] Test for musl vs libc builds
- [ ] Try `jemalloc` (There might be an issue with musl and jemalloc.)
- [ ] Consider unifying hashing and verification. Hashing and verification are implemented separately, but with some work they could be unified. This would be a good idea before pursuing performance improvements that will otherwise need to be duplicated.
- [x] Fix ultra-cringe byte-at-a-time hashing inner loop
- [ ] Experiment with buffering more data from each file
- [ ] Experiment with reading file contents in a background thread
- [ ] Experiment with async I/O
- [ ] Experiment with multiple threads
- [ ] Make new demo gif with faster speeds
- [ ] Consider mmaping files
Contributor guide
Assessment
This issue has not been assessed yet.