cat_file with start and end of gzipped file does not work.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 398
- Forks
- 181
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
Reading gzipped file using transcoding works when you use the fs.open, but not when using fs.cat_file.
Here is and example uploading 2 files, 1 plaintext, 1 gzipped, and both files are read using open, and then using cat_file:
This part works:
fs = gcsfs.GCSFileSystem(project='a')
a_file = 'same_path/a_test'
a_file_gz = 'same_path/a_test.gz'
with fs.open(a_file, 'wb') as f:
f.write(b'abcd')
with fs.open(a_file_gz, 'wb', compression='gzip', fixed_key_metadata={'content_encoding': 'gzip'}) as f:
f.write(b'abcd')
with fs.open(a_file, 'rb') as f:
assert f.read() == b'abcd'
with fs.open(a_file_gz, 'rb') as f:
assert f.read() == b'abcd'
assert bytes(fs.cat_file(a_file, 1, 3)) == b'bc'
this errors out
assert bytes(fs.cat_file(a_file_gz, 1, 3, fixed_key_metadata={'content_encoding': 'gzip'})) == b'bc'
assert bytes(fs.cat_file(a_file_gz, 1, 3)) == b'bc'
throwing this error
self = <StreamReader e=ClientPayloadError("400, message='Can not decode content-encoding: gzip'")>
n = -1
async def read(self, n: int = -1) -> bytes:
if self._exception is not None:
> raise self._exception
E aiohttp.client_exceptions.ClientPayloadError: 400, message='Can not decode content-encoding: gzip'
C:\tools\miniconda3\envs\filesystem-py39\lib\site-packages\aiohttp\streams.py:349: ClientPayloadError
my guess is because it's not passing the header in https://github.com/fsspec/gcsfs/blob/main/gcsfs/core.py#L859-L863
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read gcsfs/core.py around lines 859-863 and trace fs.cat_file's ranged read handling for objects with content_encoding gzip. Reproduce the two failing assertions from the issue, then verify both metadata variants return b'bc' without the aiohttp decoding error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100