fsspec / fsspec/gcsfs

cat_file with start and end of gzipped file does not work.

Open
#512 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

Reading gzipped file using transcoding works when you use the fs.open, but not when using fs.cat_file.
Here is and example uploading 2 files, 1 plaintext, 1 gzipped, and both files are read using open, and then using cat_file:

This part works:

fs = gcsfs.GCSFileSystem(project='a')
a_file = 'same_path/a_test'
a_file_gz = 'same_path/a_test.gz'
with fs.open(a_file, 'wb') as f:
    f.write(b'abcd')
with fs.open(a_file_gz, 'wb', compression='gzip', fixed_key_metadata={'content_encoding': 'gzip'}) as f:
    f.write(b'abcd')
with fs.open(a_file, 'rb') as f:
    assert f.read() == b'abcd'
with fs.open(a_file_gz, 'rb') as f:
    assert f.read() == b'abcd'
assert bytes(fs.cat_file(a_file, 1, 3)) == b'bc'

this errors out

assert bytes(fs.cat_file(a_file_gz, 1, 3, fixed_key_metadata={'content_encoding': 'gzip'})) == b'bc'
assert bytes(fs.cat_file(a_file_gz, 1, 3)) == b'bc'

throwing this error

self = <StreamReader e=ClientPayloadError("400, message='Can not decode content-encoding: gzip'")>
n = -1

    async def read(self, n: int = -1) -> bytes:
        if self._exception is not None:
>           raise self._exception
E           aiohttp.client_exceptions.ClientPayloadError: 400, message='Can not decode content-encoding: gzip'

C:\tools\miniconda3\envs\filesystem-py39\lib\site-packages\aiohttp\streams.py:349: ClientPayloadError

my guess is because it's not passing the header in https://github.com/fsspec/gcsfs/blob/main/gcsfs/core.py#L859-L863

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read gcsfs/core.py around lines 859-863 and trace fs.cat_file's ranged read handling for objects with content_encoding gzip. Reproduce the two failing assertions from the issue, then verify both metadata variants return b'bc' without the aiohttp decoding error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.