fsspec / fsspec/filesystem_spec

Inconsistent behaviour of `get_file` using compression with different filesystems

Open
#1,758 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I have problems getting consistent behaviours when using get_file for different filesystems when using the compression parameter. My understanding from the AbstractFilesystem implementation of that method is that kwargs should be used by the open method, but for some filesystems it fails silently.
My goal was to fetch files and decompress them on the fly: maybe there is a better suited function for this?

Minimal example:

import fsspec
import bz2
from zipfile import ZipFile

# create data file
filename = "/tmp/important_data.txt.bz2"
data = b"very important data."
with open(filename, "wb") as fd:
    fd.write(bz2.compress(data))

# open with compression
print(fsspec.open(filename, compression="infer").open().read())
# prints "b'very important data.'"

# fetch from local filesystem
fsspec.filesystem("file").get_file(filename, "/tmp/new", compression="infer")
print(open("/tmp/new", "rb").read())
# prints "b'BZh91AY&SY\x85\xf4|P\x00\x00\t\x11\x80@\x01&#\xd5  \x00"\x9e\x93i\x06\xca\x10\x00\x02\xdc\xc6\x0c\xb1\xc2\xbc\xad\x16\xc7\xc5\xdc\x91N\x14$!}\x1f\x14\x00'"

# fetch from ssh filesystem
fsspec.filesystem("ssh", host="localhost").get_file(filename, "/tmp/new", compression="infer")
print(open("/tmp/new", "rb").read())
# prints "b'BZh91AY&SY\x85\xf4|P\x00\x00\t\x11\x80@\x01&#\xd5  \x00"\x9e\x93i\x06\xca\x10\x00\x02\xdc\xc6\x0c\xb1\xc2\xbc\xad\x16\xc7\xc5\xdc\x91N\x14$!}\x1f\x14\x00'"

# fetch from zip filesystem
zfile = filename + ".zip"
with ZipFile(zfile, 'w') as zipf:
    zipf.write(filename)
of = fsspec.open("zip://" + filename + "::file://" + zfile)
of.fs.get_file(filename, "/tmp/new", compression="infer")
print(open("/tmp/new", "rb").read())
# prints "b'very important data.'"

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the AbstractFilesystem get_file implementation and run the minimal reproduction from the issue across the file, SSH, and ZIP filesystems. Compare how each handles the compression argument, then establish consistent decompressed output or document the appropriate alternative for fetching and decompressing files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.