Support for `zlib` compressed archives (`gzip`/`zip`)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 366
- Forks
- 96
- Avg merge
- 24m
- Merged PRs (30d)
- 1
Description
As mentioned here kerchunk cannot be used with zip archives that are internally compressed, because the zlib DEFLATE compression that underlies zip archives does not support random access reads.
However, I recently became aware of work by the zlib team that allows you to create sidecar index files for compressed archives that enable random reads (see this script). This functionality has been utilized by others to create a python package that enables random access read capabilities for gzip archives (which also use the DEFLATE compression algorithm).
I think it would be great to explore if these capabilities could be adapted and used by kerchunk to enable indexing of compress zip and gzip archives.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the archive-files section in docs/source/advanced.rst, then examine zlib's examples/zran.c and the indexed_gzip package referenced in the issue. Determine whether their sidecar indexing approach can be adapted to kerchunk. Done means compressed zip and gzip archives support indexing and random-access reads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100