fsspec / fsspec/filesystem_spec
`TarFileSystem`: Multiple `open()` fails with `compression`
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
cifar-10-python.tar:
tfs = fsspec.filesystem('tar', fo='cifar-10-python.tar')
for p in tfs.glob('*/data_batch_*'):
batch = tfs.open(p, 'rb')
batch.close()
cifar-10-python.tar.gz:
tfs = fsspec.filesystem('tar', fo='cifar-10-python.tar.gz', compression='gzip')
for p in tfs.glob('*/data_batch_*'):
batch = tfs.open(p, 'rb')
batch.close()
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Input In [20], in <cell line: 19>()
19 for p in tfs.glob('*/data_batch_*'):
---> 20 batch = tfs.open(p, 'rb')
21 batch.close()
File ~/.local/lib/python3.9/site-packages/fsspec/spec.py:1034, in AbstractFileSystem.open(self, path, mode, block_size, cache_options, compression, **kwargs)
1032 else:
1033 ac = kwargs.pop("autocommit", not self._intrans)
-> 1034 f = self._open(
1035 path,
1036 mode=mode,
1037 block_size=block_size,
1038 autocommit=ac,
1039 cache_options=cache_options,
1040 **kwargs,
1041 )
1042 if compression is not None:
1043 from fsspec.compression import compr
File ~/.local/lib/python3.9/site-packages/fsspec/implementations/tar.py:132, in TarFileSystem._open(self, path, mode, **kwargs)
130 else:
131 newfo = copy.copy(self.fo)
--> 132 newfo.seek(offset)
134 return TarContainedFile(newfo, self.info(path))
File /opt/conda/lib/python3.9/gzip.py:392, in GzipFile.seek(self, offset, whence)
390 elif self.mode == READ:
391 self._check_not_closed()
--> 392 return self._buffer.seek(offset, whence)
394 return self.offset
ValueError: seek of closed file
Edit: possible duplicate of https://github.com/fsspec/filesystem_spec/pull/1025#issuecomment-1263651166
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in fsspec/implementations/tar.py, especially TarFileSystem._open, and reproduce the two examples from the issue with compressed and uncompressed tar files. Check the referenced pull request discussion for related context; done means repeated opens of entries from a compressed tar file no longer fail with a closed-file seek error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100