fsspec / fsspec/filesystem_spec

[feature request] More global porcelain methods like `fsspec.listdir(...)` for basic usecases

Open
#1,860 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

Currently the user must instantiate a filesystem object manually (and thus know the filesystem name) to do analogue of os.listdir(...).

It would be nice to have more methods like existing fsspec.open(...) mimicking Python's core open(...). E.g. methods with similar invokation API as os.listdir(...), os.path.exists(...) etc (for simplifying porting existing scripts)

E.g. to be able to write generic code for simple usecases like:

# set up environment variables like AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, AWS_ENDPOINT_URL
# python ls.py s3://mybucket/

import sys, fsspec

print(fsspec.listdir(sys.argv[1]))

My current crutch:

import sys, fsspec
path = sys.argv[1]
listdir = fsspec.filesystem(path.split('://')[0] if '://' in path else 'file').ls
# from os import listdir
print(listdir(path))

# python lsfsspec.py .
# python lsfsspec.py file://.
# python lsfsspec.py s3://mybucket/

A complementing idea could be creating fsspec.Path - an analogue of pathlib.Path object accepting arbitrary paths like s3:// or github:// which would instantiate/cache filesystem object automatically and use some global configuration like environment variables, and providing the methods which pathlib.Path currently supports

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing fsspec.open(...) entry point and compare its invocation with os.listdir(...) and os.path.exists(...). Review the requested global methods and the complementary fsspec.Path idea, then define a focused scope and acceptance criteria for generic local and remote paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.