fsspec / fsspec/filesystem_spec
Inconsistent behaviour of `get_file` using compression with different filesystems
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I have problems getting consistent behaviours when using get_file for different filesystems when using the compression parameter. My understanding from the AbstractFilesystem implementation of that method is that kwargs should be used by the open method, but for some filesystems it fails silently.
My goal was to fetch files and decompress them on the fly: maybe there is a better suited function for this?
Minimal example:
import fsspec
import bz2
from zipfile import ZipFile
# create data file
filename = "/tmp/important_data.txt.bz2"
data = b"very important data."
with open(filename, "wb") as fd:
fd.write(bz2.compress(data))
# open with compression
print(fsspec.open(filename, compression="infer").open().read())
# prints "b'very important data.'"
# fetch from local filesystem
fsspec.filesystem("file").get_file(filename, "/tmp/new", compression="infer")
print(open("/tmp/new", "rb").read())
# prints "b'BZh91AY&SY\x85\xf4|P\x00\x00\t\x11\x80@\x01&#\xd5 \x00"\x9e\x93i\x06\xca\x10\x00\x02\xdc\xc6\x0c\xb1\xc2\xbc\xad\x16\xc7\xc5\xdc\x91N\x14$!}\x1f\x14\x00'"
# fetch from ssh filesystem
fsspec.filesystem("ssh", host="localhost").get_file(filename, "/tmp/new", compression="infer")
print(open("/tmp/new", "rb").read())
# prints "b'BZh91AY&SY\x85\xf4|P\x00\x00\t\x11\x80@\x01&#\xd5 \x00"\x9e\x93i\x06\xca\x10\x00\x02\xdc\xc6\x0c\xb1\xc2\xbc\xad\x16\xc7\xc5\xdc\x91N\x14$!}\x1f\x14\x00'"
# fetch from zip filesystem
zfile = filename + ".zip"
with ZipFile(zfile, 'w') as zipf:
zipf.write(filename)
of = fsspec.open("zip://" + filename + "::file://" + zfile)
of.fs.get_file(filename, "/tmp/new", compression="infer")
print(open("/tmp/new", "rb").read())
# prints "b'very important data.'"
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the AbstractFilesystem get_file implementation and run the minimal reproduction from the issue across the file, SSH, and ZIP filesystems. Compare how each handles the compression argument, then establish consistent decompressed output or document the appropriate alternative for fetching and decompressing files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100