Use rolling hash splitter (data sensitive, e.g. Buzhash or Rabin) by default instead of SizeSplitter (data insensitive)
Open
need/triage
- Dominant language
- Go
- Stars
- 316
- Forks
- 163
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 8
Description
Use the Buzhash or Rabin splitter in IPFS instead of the `SizeSplitter` by default.
A rolling hash based data sensitive splitter has a huge obvious advantage: it creates shift-resistant chunks and thus improves data deduplication and sharing.
For example `casync` is doing it too:
* http://0pointer.net/blog/casync-a-tool-for-distributing-file-system-images.html
* https://github.com/systemd/casync
* https://github.com/folbricht/desync (alternative Go implementation)
Original ticket: https://github.com/ipfs/go-ipfs-chunker/issues/13
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.