Intelligent chunking for archives

#### Type:

Enhancement.

#### Description:

A commonly cited use of IPFS is in managing archival data, however currently the process IPFS handles archives with could be improved. Take the use case of an archival utility that regularly takes a snapshot of a set of files in ZIP format. Despite many of the files being identical between the archives, even slight changes (especially in files earlier in the archive) can cause the archive to produce an entirely different hash, unnecessarily requiring complete duplication of the archive in the IPFS repo.


Pictured above is the file structure of the ZIP-64 archive format. The method to read it is to parse the final bytes of the archive to get the offset of the central directory, and then step through the directory which contains the offsets of every compressed file in the archive. If these file offsets were interpreted by IPFS as bytes to start a new chunk at, this alone would be sufficient to deduplicate the files across any number of zip archives (presuming the archives were created using the same compression options).

No files, tests, or entry points are named; begin by tracing how Kubo parses archives and chooses chunk boundaries, then assess the proposed ZIP central-directory offsets. Done would require an agreed design, implementation scope, and tests demonstrating deduplication across changed archives.

Written by the indexing model from the issue text.