fsspec / fsspec/filesystem_spec

S3 URL containing versionId is misinterpreted as glob pattern

Open
#1,439 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I am trying to load a specific version of an S3 object into a Dask dataframe as follows:

>>> import dask.dataframe as dd
>>> dd.read_parquet("s3://example-bucket/dataset/part.0.parquet?versionId=se654wJoRQcirhKoHRkN4hsFmhNwMg27")

This example returns an empty DataFrame, even though the S3 object version exists. If I omit ?versionId=..., it successfully loads the latest version.

According the s3fs docs, the versionId query parameter is supported (given that version_aware=True is set in the filesystem config).

I stepped through this in a debugger, and ended up in the expand_paths_if_needed function. This function interprets the path as a glob pattern because it contains a ? character (in ?versionId=...). This glob pattern matches nothing, since no such S3 keys exist, and an empty list of paths is returned.

I'm not sure if this needs to be fixed in ffspec or dask. Versioned S3 objects seem to be supported by s3fs, but the higher-level fsspec doesn't expect query parameters in URLs.

Maybe this use case wasn't intended. Do you think it's feasible to support it?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fsspec/core.py at expand_paths_if_needed, using the reported S3 URL with a versionId query parameter to reproduce the empty path result. Trace how the question mark is classified as a glob pattern; done means versioned S3 URLs reach s3fs without glob expansion and load the requested object version.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.