fsspec / fsspec/filesystem_spec

AbstractFileSystem is not abstract

Open
#1,446 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I'm trying to learn more about how fsspec works (so I can better understand the IO part of the Pangeo stack), but I'm confused by the structure of fsspec/filesystem_spec.

Despite the name, it seems AbstractFileSystem is not an abstract base class (nor is AbstractBufferedFile) - I'm able to import and instantiate it directly. It contains lots of actual code that does things.

Neither is there a typing.Protocol that subclasses could conform to that could play the same role. (In fact there is no typing at all, which seems like an issue for a project whose purpose is to ensure interface compatibility. Wouldn't it help solve issues like https://github.com/fsspec/filesystem_spec/issues/1411?)

Finally I can't find a written spec document either, at least not one that conforms to RFC 2119 like the Zarr Spec does for example. The fsspec documentation just points me to fsspec/spec.py.

As a new user / potential developer using fsspec, how am I supposed to know which parts of AbstractFileSystem are implementation details that can be overridden, and which are actual requirements? Is there some reason why this lack of a distinct specification doesn't cause problems in practice? Apologies if I have missed something here, I am new to this whole area.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with fsspec/spec.py and the documentation section on the package structure to compare the described AbstractFileSystem and AbstractBufferedFile roles. Review the linked typing issue #625 and compatibility issue #1411 for related context. Done would require a clear decision about the interface contract and documentation of which members are requirements versus implementation details.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.