allenai / allenai/cached_path

Add support for zstandard, bzip2, and lzma compressed tar files

Open
#259 0 comments 0 reactions 0 assignees View on GitHub
Feature request
Dominant language
Python
Stars
47
Forks
22
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
Currently this library only supports `*.tar.gz` (gzip) files. This needs to be extended to also support `*.tar.zst` (zstandard), `*.tar.bz2` (bzip2), and `*.tzr.xz` (lzma) files

**Describe the solution you'd like**
Automatically decompress the compressed tar-file using the appropriate decompressor. In Python 3.14+ this can be accomplished using `mode` kwarg in `tarfile.open` ([link](https://docs.python.org/3.14/library/tarfile.html#tarfile.open)). In older Python versions, we will need special handling for zstandard compressed tar files (using the standalone [python-zstandard](https://github.com/indygreg/python-zstandard) library) since `tarfile.open` doesn't yet support zstandard.

**Describe alternatives you've considered**
None

**Additional context**
None

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.