fsspec / fsspec/filesystem_spec
lost "seekable" parameter while load a zip file on HDFS
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I'm using fsspec to load a zip file which store in HDFS, but encounter a problem
python3.9/fsspec/implementations/zip.py", line 54, in __init__
self.zip = zipfile.ZipFile(self.fo, mode=mode)
File "python3.9/zipfile.py", line 1257, in __init__
self._RealGetContents()
File "python3.9/zipfile.py", line 1322, in _RealGetContents
raise BadZipFile("File is not a zip file")
this is because of the file object is not seekable:
File "/opt/miniconda2/lib/python3.9/zipfile.py", line 1320, in _RealGetContents
endrec = _EndRecData(fp)
File "/opt/miniconda2/lib/python3.9/zipfile.py", line 263, in _EndRecData
fpin.seek(0, 2)
File "pyarrow/io.pxi", line 308, in pyarrow.lib.NativeFile.seek
File "pyarrow/io.pxi", line 206, in pyarrow.lib.NativeFile.get_random_access_file
File "pyarrow/io.pxi", line 235, in pyarrow.lib.NativeFile._assert_seekable
OSError: only valid on seekable files
I look into fsspec implement and found a problem or maybe a bug
let's see call stack first

in the bottom of the call stack, _open function in implementations/arrow.py:156 , here is the seekable parameter and default value is False;
and fourth call frame from the bottom, __enter__ function in core.py:102, you lost this parameter.
once I change _open function seekable parameter default to True (just for verify), every thing goes well
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting implementations/arrow.py around _open and core.py around enter, following the seekable argument through the call stack shown in the report. Reproduce loading a ZIP file from HDFS and verify that the seekability requirement is preserved so the archive opens successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100