fsspec / fsspec/gcsfs

read() returns incomplete file? (not clear on documentation)

Open
#233 12 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

Hello,

This is my first time using the library, so please apologize me if I'm missing something obvious.

I'm trying to read a .csv.gz file (quite small, less than 100kb) following the documentation (https://gcsfs.readthedocs.io/en/latest/#examples), by doing:

fs = gcsfs.GCSFileSystem(project=project_id, token=service_key_path)
uri = bucket_name + '/' + filename
with fs.open(uri, 'rb') as f:
    content = f.read()

Here, I expect content to have the entire file, but it has only 168 rows (out of 946 in total). Reading the same file with google-cloud-storage works fine.

storage_client = storage.Client(credentials=credentials, project=project_id)
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(filename)
content = blob.download_as_string()

What am I missing?
Thanks

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the documented gcsfs example and the fs.open(..., 'rb').read() entry point described in the report, then compare its result with google-cloud-storage's download_as_string(). Reproduce the .csv.gz case and determine whether the incomplete content is a library bug or a documentation issue; done means the cause and expected behavior are established and covered by a clear fix or documentation update.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.