huggingface / huggingface/hf_transfer
More advanced pipelining for hf_transfer to increase download speed even further
- Dominant language
- Rust
- Stars
- 580
- Forks
- 41
- Avg merge
- 7m
- Merged PRs (30d)
- 1
Description
**Is your feature request related to a problem? Please describe.**
hf_transfer is very fast for individual files, but for models with many split files, it's not quite as fast as it could be.
**Describe the solution you'd like**
Currently, it seems like hf_transfer is invoked serially for each file, starting over from scratch with a new set of connections. That results in a timeline similar to this:
To fully maximize the download speed, what it should do instead is use a shared pool of connections that is reused between small files and chunks of multiple larger files in parallel:
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.