borgbackup / borgbackup/borg

file type based chunking

Open
#1,005 14 comments 0 reactions 0 assignees View on GitHub
c: chunker cmd: create enhancement
Dominant language
Python
Stars
13.7k
Forks
875
Avg merge
11h 15m
Merged PRs (30d)
192

Description

depending on the file type (filename extension), we could use different chunkers.

e.g. for uncompressed tar files, we could split at file boundaries to better deduplicate different tars that share a lot of files.

Ideas by @dragetd (I removed the parts looking into compressed formats or into filesystem images - going there would involve exact reproduction of compressed archives / of the filesystem image which is not trivial and not in the scope of this ticket):

It could chunk along boundaries of files not only within TARs, [...] header information of media-files that might change more likely compared to the data-stream; [...] EXIF-tags and some raw image formats might contain previews that might fit neatly into a chunk… etc. etc.

I could see some real benefits. And funky heuristics and support for certain filetypes could be added incrementally without harming any compatibility. And yet, it would require a lot of work and knowledge of all the various formats… and it needs to be crafted carefully so the parser will robustly only output suggested chunk-sizes to the chunker, and not be prone to security issues when trying to parse dozens of filetypes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.