fsspec / fsspec/filesystem_spec

lost "seekable" parameter while load a zip file on HDFS

Open
#1,189 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I'm using fsspec to load a zip file which store in HDFS, but encounter a problem

  python3.9/fsspec/implementations/zip.py", line 54, in __init__
    self.zip = zipfile.ZipFile(self.fo, mode=mode)
  File "python3.9/zipfile.py", line 1257, in __init__
    self._RealGetContents()
  File "python3.9/zipfile.py", line 1322, in _RealGetContents
    raise BadZipFile("File is not a zip file")

this is because of the file object is not seekable:

  File "/opt/miniconda2/lib/python3.9/zipfile.py", line 1320, in _RealGetContents
    endrec = _EndRecData(fp)
  File "/opt/miniconda2/lib/python3.9/zipfile.py", line 263, in _EndRecData
    fpin.seek(0, 2)
  File "pyarrow/io.pxi", line 308, in pyarrow.lib.NativeFile.seek
  File "pyarrow/io.pxi", line 206, in pyarrow.lib.NativeFile.get_random_access_file
  File "pyarrow/io.pxi", line 235, in pyarrow.lib.NativeFile._assert_seekable
OSError: only valid on seekable files

I look into fsspec implement and found a problem or maybe a bug
let's see call stack first

SeaTalk_IMG_1676617873

in the bottom of the call stack, _open function in implementations/arrow.py:156 , here is the seekable parameter and default value is False;
and fourth call frame from the bottom, __enter__ function in core.py:102, you lost this parameter.

once I change _open function seekable parameter default to True (just for verify), every thing goes well

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting implementations/arrow.py around _open and core.py around enter, following the seekable argument through the call stack shown in the report. Reproduce loading a ZIP file from HDFS and verify that the seekability requirement is preserved so the archive opens successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.