Add a buzhash preset with a smaller chunk-size target
Open
kind/enhancement
- Dominant language
- Go
- Stars
- 17.1k
- Forks
- 3.2k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 11
Description
With rabin we had a lot of flexibility regarding the chunk size it's been producing. With buzhash, we only get a one-keyword-preset with around 256 K size.
While I get the idea behind it, 256 K chunks are for some data just too large to create sensible deduplication. An example would be SQL dumps and VM images.
It would be nice to have an additional keyword for a smaller target chunk-size, like 8 K or 16 K to get a better deduplication ratio.
Contributor guide
Assessment
This issue has not been assessed yet.