new feature: Implement fsspec.AsyncFileSystem in OpenDAL Python Bindings
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
This feature proposes implementing the fsspec.AsyncFileSystem interface directly within the opendal Python bindings. This would allow OpenDAL to serve as a backend-compatible filesystem for widely-used Python libraries such as **DuckDB, Polars, PyArrow**, and others that rely on fsspec for abstracted filesystem access.
An optional implementation of pyarrow.fs.FSSpecHandler could also be provided for better integration with PyArrow’s native filesystem API.
### Problem and Solution
**Problem:**
Many Python-based data tools (DuckDB, Polars, PyArrow, etc.) rely on fsspec for interacting with storage backends. While OpenDAL offers a unified interface for accessing remote and local storage, it currently does not expose an fsspec-compatible interface, making it non-trivial to use OpenDAL directly with these tools.
There exists an external experimental project ([opendalfs](https://github.com/fsspec/opendalfs)) to bridge this gap via Rust, but it duplicates functionality already available in OpenDAL’s Python bindings and adds overhead in terms of maintenance and packaging.
**Solution:**
By implementing fsspec.AsyncFileSystem inside OpenDAL’s Python bindings (possibly as an optional extra), OpenDAL can natively power fsspec-based workflows without reinventing the wheel.
This solution:
- Allows immediate compatibility with popular Python data tools
- Avoids redundant Rust+Python implementations (as in opendalfs)
- Keeps maintenance, versioning, and testing centralised in the OpenDAL repo
- Reduces friction for users who want to use OpenDAL as a unified data access layer in Python
- Boosts real-world adoption of OpenDAL through seamless integration with existing ecosystems
### Additional Context
This proposal was discussed in detail on a GitHub conversation in the OpenDAL repository:
📍 Reference Thread: [(rust) How to use opendal to load data into DuckDB or Polars LazyFrame? #5972](https://github.com/apache/opendal/discussions/5972)
- Explored how current limitations in the Rust APIs of Polars and DuckDB make direct integration difficult.
- Highlighted that fsspec is already widely supported in Python bindings for these tools.
- Suggested that instead of building a standalone solution like opendalfs, integrating it directly into OpenDAL’s Python bindings would lead to better long-term maintainability and adoption.
- Aligns with the direction acknowledged by maintainers like @Xuanwo.
### Are you willing to contribute to the development of this feature?
- [x] Yes, I am willing to contribute to the development of this feature.
Contributor guide
Research direction
No implementation files or tests are named. Start by reviewing the Python bindings and the referenced discussion #5972, then define the fsspec.AsyncFileSystem scope and compatibility expectations for DuckDB, Polars, and PyArrow; done means the bindings provide the agreed interface, with FSSpecHandler only if included in scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100