ipfs / ipfs/kubo

enhancement: tarsnap chunker / importer (Content defined, polynomial, fast converging chunking)

Open
#3,603 1 comment 2 reactions 0 assignees View on GitHub
help wanted kind/enhancement
Dominant language
Go
Stars
17.1k
Forks
3.2k
Avg merge
3d 18h
Merged PRs (30d)
11

Description

#### Version information:
go-ipfs version: 0.4.4

#### Type: Feature, Enhancement

#### Priority: P4

#### Area: Tools, Importer

#### Description:
Better suited for maximizing deduplication ratio then current Rabin chunker.
Using smaller chunks with faster convergence yields greater space savings, and the benefit depending on dataset can be great in comparison to Rabin.

The mean chunk size used by tarsnap is 64k.

Source:
https://github.com/Tarsnap/tarsnap/blob/master/tar/multitape/chunkify.h
https://github.com/Tarsnap/tarsnap/blob/master/tar/multitape/chunkify.c

Related:
https://moinakg.wordpress.com/2012/11/11/inside-content-defined-chunking-in-pcompress/
https://moinakg.wordpress.com/2012/11/15/inside-content-defined-chunking-in-pcompress-part-2/

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.