file type based chunking
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 875
- Avg merge
- 11h 15m
- Merged PRs (30d)
- 192
Description
depending on the file type (filename extension), we could use different chunkers.
e.g. for uncompressed tar files, we could split at file boundaries to better deduplicate different tars that share a lot of files.
Ideas by @dragetd (I removed the parts looking into compressed formats or into filesystem images - going there would involve exact reproduction of compressed archives / of the filesystem image which is not trivial and not in the scope of this ticket):
It could chunk along boundaries of files not only within TARs, [...] header information of media-files that might change more likely compared to the data-stream; [...] EXIF-tags and some raw image formats might contain previews that might fit neatly into a chunk… etc. etc.
I could see some real benefits. And funky heuristics and support for certain filetypes could be added incrementally without harming any compatibility. And yet, it would require a lot of work and knowledge of all the various formats… and it needs to be crafted carefully so the parser will robustly only output suggested chunk-sizes to the chunker, and not be prone to security issues when trying to parse dozens of filetypes.
Contributor guide
Assessment
This issue has not been assessed yet.