casey / casey/intermodal

Improve hashing and verification performance

Open
#26 12 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue
Dominant language
Rust
Stars
659
Forks
36
Avg merge
18m
Merged PRs (30d)
2

Description

The speed of torrent creation and verification could use some work. I picked SHA1 and MD5 implementations basically at random, and there are lots of potential performance bottlenecks in how I hash and verify pieces.

Also, #440 reports performance problems when linking against musl on Linux. If allocation, I/O, or threading is slow with musl, then general performance improvements might cause less allocation and/or I/O (I don't do any threading) which might lessen the performance impact of linking against musl.

First thing will be to set up repeatable benchmarking, so there's stable baseline to compare against. Next there are a bunch of things that are worth trying.

- [x] Split crate into binary and library, to make benchmarking possible
- [x] Set up benchmarking
- [ ] Set up flamegraph generation
- [ ] Tests which exercise incomplete reads and uneven files, and probably some known good infohashes of large files and directories of files. (They can be generated with an algorithm, so they don't need to be saved in the repo.)
- [ ] Test for musl vs libc builds
- [ ] Try `jemalloc` (There might be an issue with musl and jemalloc.)
- [ ] Consider unifying hashing and verification. Hashing and verification are implemented separately, but with some work they could be unified. This would be a good idea before pursuing performance improvements that will otherwise need to be duplicated.
- [x] Fix ultra-cringe byte-at-a-time hashing inner loop
- [ ] Experiment with buffering more data from each file
- [ ] Experiment with reading file contents in a background thread
- [ ] Experiment with async I/O
- [ ] Experiment with multiple threads
- [ ] Make new demo gif with faster speeds
- [ ] Consider mmaping files

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.