fsspec / fsspec/filesystem_spec

`TarFileSystem`: Multiple `open()` fails with `compression`

Open
#1,064 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

cifar-10-python.tar:

tfs = fsspec.filesystem('tar', fo='cifar-10-python.tar')

for p in tfs.glob('*/data_batch_*'): 
    batch = tfs.open(p, 'rb')
    batch.close()

cifar-10-python.tar.gz:

tfs = fsspec.filesystem('tar', fo='cifar-10-python.tar.gz', compression='gzip')

for p in tfs.glob('*/data_batch_*'): 
    batch = tfs.open(p, 'rb')
    batch.close()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [20], in <cell line: 19>()
     19 for p in tfs.glob('*/data_batch_*'): 
---> 20     batch = tfs.open(p, 'rb')
     21     batch.close()

File ~/.local/lib/python3.9/site-packages/fsspec/spec.py:1034, in AbstractFileSystem.open(self, path, mode, block_size, cache_options, compression, **kwargs)
   1032 else:
   1033     ac = kwargs.pop("autocommit", not self._intrans)
-> 1034     f = self._open(
   1035         path,
   1036         mode=mode,
   1037         block_size=block_size,
   1038         autocommit=ac,
   1039         cache_options=cache_options,
   1040         **kwargs,
   1041     )
   1042     if compression is not None:
   1043         from fsspec.compression import compr

File ~/.local/lib/python3.9/site-packages/fsspec/implementations/tar.py:132, in TarFileSystem._open(self, path, mode, **kwargs)
    130 else:
    131     newfo = copy.copy(self.fo)
--> 132 newfo.seek(offset)
    134 return TarContainedFile(newfo, self.info(path))

File /opt/conda/lib/python3.9/gzip.py:392, in GzipFile.seek(self, offset, whence)
    390 elif self.mode == READ:
    391     self._check_not_closed()
--> 392     return self._buffer.seek(offset, whence)
    394 return self.offset

ValueError: seek of closed file

Edit: possible duplicate of https://github.com/fsspec/filesystem_spec/pull/1025#issuecomment-1263651166

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fsspec/implementations/tar.py, especially TarFileSystem._open, and reproduce the two examples from the issue with compressed and uncompressed tar files. Check the referenced pull request discussion for related context; done means repeated opens of entries from a compressed tar file no longer fail with a closed-file seek error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.