keep prefix after glob?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 305
- Avg merge
- 22h 37m
- Merged PRs (30d)
- 4
Description
I often use glob to get a list of files. I may loop over this to do similar operations over a file. It'll be nice if I could keep the prefix e.g. (s3://) instead of adding that back in myself:
import s3fs
fs = s3fs.S3FileSystem()
files = fs.glob("s3://bucket/YYYYMM*")
# files = [bucket/path/YYYYMM01, bucket/path/YYYYMM02] ...
Would like
# files = [s3://bucket/path/YYYYMM01, s3://bucket/path/YYYYMM02] ...
Easy to add a user but common enough that it would be nice if I could avoid this step. e.g. have a keep_prefix=True kwarg?
Current work arounds
for file in files:
pd.read_parquet("s3://" + file)
or
files = ["s3://" + file for file in files]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the s3fs glob implementation used by fs.glob("s3://bucket/YYYYMM*") and check how returned paths are assembled. The change is done when an optional keep_prefix=True form returns paths retaining the s3:// prefix while existing output remains unchanged; add or update focused tests for both forms.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100