fsspec / fsspec/filesystem_spec

UnsupportedOperation error with ZipFileSystem using append mode

Open
#1,449 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

fsspec.implementations.zip.ZipFileSystem seems to have an issue with append mode 'a'. I assume, opening in mode 'a' allows for adding new Zip-entries to an existing Zip archive. At least this is what I need. While opening in append mode first seems to work - I can write to the archive - it will fail once the filesystem is closed.

While using mode 'w' works and creates an empty archive like so:

fs = fsspec.filesystem("zip", fo="test-w.zip", mode="w")
# write stuff here or do nothing...
fs = None  # <-- force garbage collection of fs object

An error is raised when using mode 'a' once the filesystem is auto-closed by setting the reference to None or by calling fs.close() directly:

fs = fsspec.filesystem("zip", fo="test-a.zip", mode="a")
# write stuff here or do nothing...
fs = None   # <-- error, also for fs.close()

Traceback:

Exception ignored in: <function ZipFileSystem.__del__ at 0x7fa07d5e4fe0>
Traceback (most recent call last):
  File ".../lib/python3.11/site-packages/fsspec/implementations/zip.py", line 73, in __del__
    self.close()
  File ".../lib/python3.11/site-packages/fsspec/implementations/zip.py", line 78, in close
    self.zip.close()
  File ".../lib/python3.11/zipfile.py", line 1892, in close
    self._write_end_record()
  File ".../lib/python3.11/zipfile.py", line 1989, in _write_end_record
    self.fp.truncate()
io.UnsupportedOperation: truncate

Reproducible on Linux and Windows.

Note, Python docs on zipfile.ZipFile state: If mode is 'a' and the file does not exist at all, it is created. If mode is 'r' or 'a', the file should be seekable.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with fsspec/implementations/zip.py, especially ZipFileSystem.del and close, and reproduce the provided append-mode example on close. Compare the append and write paths and verify that closing an archive opened with mode 'a' no longer raises io.UnsupportedOperation: truncate while still allowing entries to be added.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.