fsspec / fsspec/s3fs

keep prefix after glob?

Open
#509 3 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
305
Avg merge
22h 37m
Merged PRs (30d)
4

Description

I often use glob to get a list of files. I may loop over this to do similar operations over a file. It'll be nice if I could keep the prefix e.g. (s3://) instead of adding that back in myself:

import s3fs
fs = s3fs.S3FileSystem()

files = fs.glob("s3://bucket/YYYYMM*")
# files = [bucket/path/YYYYMM01, bucket/path/YYYYMM02] ...

Would like

# files = [s3://bucket/path/YYYYMM01, s3://bucket/path/YYYYMM02] ...

Easy to add a user but common enough that it would be nice if I could avoid this step. e.g. have a keep_prefix=True kwarg?

Current work arounds

for file in files:
    pd.read_parquet("s3://" + file)

or

files = ["s3://" + file for file in files]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the s3fs glob implementation used by fs.glob("s3://bucket/YYYYMM*") and check how returned paths are assembled. The change is done when an optional keep_prefix=True form returns paths retaining the s3:// prefix while existing output remains unchanged; add or update focused tests for both forms.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.