borgbackup / borgbackup/borg

create separate project for chunkers

Open
#9,280 6 comments 0 reactions 0 assignees View on GitHub
c: chunker
Dominant language
Python
Stars
13.7k
Forks
875
Avg merge
11h 15m
Merged PRs (30d)
192

Description

- evaluate modern chunking algorithms (fastcdc, ultracdc, ...)
- implement chunking algorithms for usage within borg (chunker written in Cython or from suitable, well-maintained existing external package, tests, interface with existing borg code)
- evaluate deduplication and throughput performance
- security: avoid fingerprinting and other attacks on the chunker
- rip out the borg.chunkers package from borg and create a borgchunk separate project with own tests and documentation

This task does not require much knowledge about the borg code base, but will require evaluating research papers, Cython coding, doing practical experiments, etc. - some math/crypto/security/algorithms background is useful here.

I recently converted our buzhash(32) code from C to Cython, then added a quite similar buzhash64 chunker (also Cython). So all code now is in Cython (some also in Python), no C anymore.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.