fsspec / fsspec/filesystem_spec
Getting a generator or iterator instead of lists of OpenFile objects or addresses.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Functions like fsspec.open_files or FileSystem.ls return list-like objects when run on directories or with glob patterns. This has two main drawbacks:
- The functions only return once the entire directory has been listed. When listing cloud buckets with millions of entries, this can take many minutes. It leads to:
- Higher failure risk due to long runtimes.
- No way to add user feedback in the meantime, e.g. a progress bar.
- Can't start processing the first files found while the rest are still being listed.
- All OpenFile objects or addresses are kept in memory at once.
Is it possible to get a generator or iterator instead of a list? I'm particularly interested in support for local, s3fs, and gcsfs.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing fsspec.open_files and FileSystem.ls for directory and glob handling, then compare the local, s3fs, and gcsfs implementations. Define how iterator behavior should work across these backends, including when results become available and what completion looks like, before identifying the tests needed to verify it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, gcp, python
- Domain
- backend-api-design, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100