fsspec / fsspec/filesystem_spec
[feature request] More global porcelain methods like `fsspec.listdir(...)` for basic usecases
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
Currently the user must instantiate a filesystem object manually (and thus know the filesystem name) to do analogue of os.listdir(...).
It would be nice to have more methods like existing fsspec.open(...) mimicking Python's core open(...). E.g. methods with similar invokation API as os.listdir(...), os.path.exists(...) etc (for simplifying porting existing scripts)
E.g. to be able to write generic code for simple usecases like:
# set up environment variables like AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, AWS_ENDPOINT_URL
# python ls.py s3://mybucket/
import sys, fsspec
print(fsspec.listdir(sys.argv[1]))
My current crutch:
import sys, fsspec
path = sys.argv[1]
listdir = fsspec.filesystem(path.split('://')[0] if '://' in path else 'file').ls
# from os import listdir
print(listdir(path))
# python lsfsspec.py .
# python lsfsspec.py file://.
# python lsfsspec.py s3://mybucket/
A complementing idea could be creating fsspec.Path - an analogue of pathlib.Path object accepting arbitrary paths like s3:// or github:// which would instantiate/cache filesystem object automatically and use some global configuration like environment variables, and providing the methods which pathlib.Path currently supports
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing fsspec.open(...) entry point and compare its invocation with os.listdir(...) and os.path.exists(...). Review the requested global methods and the complementary fsspec.Path idea, then define a focused scope and acceptance criteria for generic local and remote paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100