meta-pytorch / meta-pytorch/data
FSSpecFileLister slow warmup with many files
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🚀 The feature
It would be great if FSSpecFileLister could iteratively list files instead of preloading them all.
Motivation, pitch
With a cloud provider, listing all the files in a directory is slow and expensive. In my GCS dir with 200k files FSSpecFileLister takes forever to return a single url.
It would be great if FSSpecFileLister could iteratively list files instead of preloading them all.
Alternatives
No response
Additional context
fsspec seems to support returning a generator via walk().
https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.walk
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the FSSpecFileLister implementation and its callers. Compare its current listing behavior with fsspec's walk() generator API linked in the issue. Done means the lister can yield an initial URL without preloading all files, while preserving the expected listing behavior for large directories.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100