ipfs / ipfs/boxo

Use rolling hash splitter (data sensitive, e.g. Buzhash or Rabin) by default instead of SizeSplitter (data insensitive)

Open
#355 1 comment 0 reactions 0 assignees View on GitHub
need/triage
Dominant language
Go
Stars
316
Forks
163
Avg merge
4d 12h
Merged PRs (30d)
8

Description

Use the Buzhash or Rabin splitter in IPFS instead of the `SizeSplitter` by default.
A rolling hash based data sensitive splitter has a huge obvious advantage: it creates shift-resistant chunks and thus improves data deduplication and sharing.

For example `casync` is doing it too:
* http://0pointer.net/blog/casync-a-tool-for-distributing-file-system-images.html
* https://github.com/systemd/casync
* https://github.com/folbricht/desync (alternative Go implementation)

Original ticket: https://github.com/ipfs/go-ipfs-chunker/issues/13

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.