fsspec / fsspec/filesystem_spec
S3 URL containing versionId is misinterpreted as glob pattern
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I am trying to load a specific version of an S3 object into a Dask dataframe as follows:
>>> import dask.dataframe as dd
>>> dd.read_parquet("s3://example-bucket/dataset/part.0.parquet?versionId=se654wJoRQcirhKoHRkN4hsFmhNwMg27")
This example returns an empty DataFrame, even though the S3 object version exists. If I omit ?versionId=..., it successfully loads the latest version.
According the s3fs docs, the versionId query parameter is supported (given that version_aware=True is set in the filesystem config).
I stepped through this in a debugger, and ended up in the expand_paths_if_needed function. This function interprets the path as a glob pattern because it contains a ? character (in ?versionId=...). This glob pattern matches nothing, since no such S3 keys exist, and an empty list of paths is returned.
I'm not sure if this needs to be fixed in ffspec or dask. Versioned S3 objects seem to be supported by s3fs, but the higher-level fsspec doesn't expect query parameters in URLs.
Maybe this use case wasn't intended. Do you think it's feasible to support it?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in fsspec/core.py at expand_paths_if_needed, using the reported S3 URL with a versionId query parameter to reproduce the empty path result. Trace how the question mark is classified as a glob pattern; done means versioned S3 URLs reach s3fs without glob expansion and load the requested object version.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100