ipfs / ipfs/kubo

Add a buzhash preset with a smaller chunk-size target

Open
#7,966 4 comments 0 reactions 0 assignees View on GitHub
kind/enhancement
Dominant language
Go
Stars
17.1k
Forks
3.2k
Avg merge
3d 18h
Merged PRs (30d)
11

Description

With rabin we had a lot of flexibility regarding the chunk size it's been producing. With buzhash, we only get a one-keyword-preset with around 256 K size.

While I get the idea behind it, 256 K chunks are for some data just too large to create sensible deduplication. An example would be SQL dumps and VM images.

It would be nice to have an additional keyword for a smaller target chunk-size, like 8 K or 16 K to get a better deduplication ratio.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.