fsspec / fsspec/filesystem_spec
Provide base test suite for implementators
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 490
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 38
Description
I think the fsspec ecosystem would benefit from a set of "base" tests defined in fsspec that can be used by downstream projects like s3fs and adlfs. This would help iron out the inconsistencies between these libraries, especially around behaviors with trailing /, exceptions, etc.
Pandas has used this with extension types, which are expected to implement the extension array interface. See https://pandas.pydata.org/docs/development/extending.html#testing-extension-arrays for more, but the basic idea is to use a form of property-based testing. fsspec defines the expected behavior as a test, in a way that only uses the public fsspec interface. Then implementors would define pytest fixtures to provide the filesystem with some known properties (e.g. certain files exist).
@pytest.fixture
def fs():
"""Fixture defining the filesystem with these files:"""
...
class TestFilesystem:
def test_ls_raises_filenotfound(self, fs):
with pytest.raises(FileNotFoundError):
fs.ls("/not/a/key")
https://github.com/pandas-dev/pandas/blob/master/pandas/tests/extension/conftest.py and https://github.com/pandas-dev/pandas/blob/master/pandas/tests/extension/base/__init__.py provide some examples.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the pandas extension-testing examples linked in the issue, especially conftest.py and base/init.py, then compare them with fsspec's public filesystem interface. Define the scope of shared behavioral tests and the fixture contract for downstream filesystems. Done means downstream projects can reuse the base suite to check behaviors such as trailing slashes and missing-path exceptions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100