fsspec / fsspec/gcsfs

Generation ID ignored when reading from gcs

Open
#446 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

Hi, I am having an issue reading specific generation/versions of files from gcs.

For instance, in a bucket with versioning enabled:

# old
gs://mock-bucket/something.csv#123456789
# new
gs://mock-bucket/something.csv#999999999

when trying to read gs://mock-bucket/something.csv#123456789, it will read gs://mock-bucket/something.csv#999999999 without any warning or error message.

I am happy to make a contribution! let me know!

To reproduce:

  • Create a gcs bucket with version enabled.
  • Upload two file with different content to the bucket with the same key.
  • This will generate 2 objects with same key but they will have different generation id
  • use ffspec to read the object with older generation id
    • ie:
    from ffspec import open as f_open
    
    with f_open.read('gs://mock-bucket/something.csv#123456789', 'r') as f:
        ....   
    
    
  • ffspec will only read ignore the generation id and only read the latest files.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the fsspec.open entry point for gs:// URLs and how the generation fragment is handled. Reproduce the case with a versioned Google Cloud Storage bucket; done means reading an older generation returns that content while requests without a generation still read the latest object.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.