apache / apache/opendal

new feature: Implement fsspec.AsyncFileSystem in OpenDAL Python Bindings

Open
#6,249 0 comments 10 reactions 0 assignees View on GitHub
bindings/python enhancement releases-note/feat
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

This feature proposes implementing the fsspec.AsyncFileSystem interface directly within the opendal Python bindings. This would allow OpenDAL to serve as a backend-compatible filesystem for widely-used Python libraries such as **DuckDB, Polars, PyArrow**, and others that rely on fsspec for abstracted filesystem access.

An optional implementation of pyarrow.fs.FSSpecHandler could also be provided for better integration with PyArrow’s native filesystem API.

### Problem and Solution

**Problem:**

Many Python-based data tools (DuckDB, Polars, PyArrow, etc.) rely on fsspec for interacting with storage backends. While OpenDAL offers a unified interface for accessing remote and local storage, it currently does not expose an fsspec-compatible interface, making it non-trivial to use OpenDAL directly with these tools.

There exists an external experimental project ([opendalfs](https://github.com/fsspec/opendalfs)) to bridge this gap via Rust, but it duplicates functionality already available in OpenDAL’s Python bindings and adds overhead in terms of maintenance and packaging.

**Solution:**

By implementing fsspec.AsyncFileSystem inside OpenDAL’s Python bindings (possibly as an optional extra), OpenDAL can natively power fsspec-based workflows without reinventing the wheel.

This solution:

- Allows immediate compatibility with popular Python data tools
- Avoids redundant Rust+Python implementations (as in opendalfs)
- Keeps maintenance, versioning, and testing centralised in the OpenDAL repo
- Reduces friction for users who want to use OpenDAL as a unified data access layer in Python
- Boosts real-world adoption of OpenDAL through seamless integration with existing ecosystems

### Additional Context

This proposal was discussed in detail on a GitHub conversation in the OpenDAL repository:

📍 Reference Thread: [(rust) How to use opendal to load data into DuckDB or Polars LazyFrame? #5972](https://github.com/apache/opendal/discussions/5972)

- Explored how current limitations in the Rust APIs of Polars and DuckDB make direct integration difficult.
- Highlighted that fsspec is already widely supported in Python bindings for these tools.
- Suggested that instead of building a standalone solution like opendalfs, integrating it directly into OpenDAL’s Python bindings would lead to better long-term maintainability and adoption.
- Aligns with the direction acknowledged by maintainers like @Xuanwo.

### Are you willing to contribute to the development of this feature?

- [x] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

No implementation files or tests are named. Start by reviewing the Python bindings and the referenced discussion #5972, then define the fsspec.AsyncFileSystem scope and compatibility expectations for DuckDB, Polars, and PyArrow; done means the bindings provide the agreed interface, with FSSpecHandler only if included in scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.