containers / containers/zstd-chunked-rs
pull example: consider two optimizations
- Dominant language
- Rust
- Stars
- 12
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
The [pull example](https://github.com/containers/zstd-chunked-rs/blob/main/examples/pull.rs) shows how we can do an incremental download of a zstd:chunked image backed by a cache directory.
It's fairly naive about performing the work requested of it. We might make a couple of changes to make it better:
1. We might have multiple chunks that have the same checksum value. This might even occur between different layers that we're downloading concurrently. With the current code, if we get lucky then we'll not try to attempt the downloads concurrently, and by the time the second copy starts, it'll see the file that the first copy already left in the cache, but that might change if...
2. We might change our approach to planning the download to first scan for all of the chunks in the cache, filter the ones we have, and then attempt to sort the remaining file ranges and notice which ranges are "close" (ie: separated by a single compressed tar header). In practice, the tar header frames compress to a small amount of data, often less than 100 bytes, which is worth "wasting" if it means we get to do one less HTTP request (ie: by combining the two ranges). In that case we'll almost certainly want "notice" multiple ranges with the same checksum *before* we do the merging. In the "cold start" case we might end up combining the entire file into a single range, but this raises another question: downloading multiple ranges in parallel has performance advantages (also in terms of being able to parallelize what we save to disk), so we definitely don't want to serialize everything.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.