apache / apache/opendal

new feature: Split python binding in seperate packages

Open
#4,939 3 comments 0 reactions 0 assignees View on GitHub
bindings/python enhancement
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

The OpenDAL Python binding now releases all its services in a single package, which makes it difficult to use and extend. I propose splitting it into separate packages, similar to what I've done with [`opendalfs`](https://github.com/fsspec/opendalfs).

```mermaid
graph TD;
opendalfs.OpendalFileSystem -- import --> MemoryFileSystem;
opendalfs.OpendalFileSystem -- import --> S3FileSystem;
opendalfs.OpendalFileSystem -- import --> FsFileSystem;
opendalfs.OpendalFileSystem -- import --> ...FileSystem;
MemoryFileSystem -- use --> opendalfs-core;
S3FileSystem -- use --> opendalfs-core;
FsFileSystem -- use --> opendalfs-core;
...FileSystem -- use --> opendalfs-core;
opendalfs-core -- use -->opendal["Apache OpenDAL"];
```

### Problem and Solution

OpenDAL Python is large yet still doesn't cover all the services users need.

We can divide it into multiple packages, making `opendal` a virtual meta-package that only provides the Python API and imports the correct service on demand when needed.

For example, as I showed up in `opendalfs`:

https://github.com/fsspec/opendalfs/blob/e19d28eb9f82e285685f91b3c80805146759b7d7/opendalfs/fs.py#L8-L25

```python
def __init__(self, scheme, *args, **kwargs):
super().__init__(*args, **kwargs)

try:
# Load the module dynamically based on scheme
module = importlib.import_module(f"opendalfs_service_{scheme}")
# Get the file system class based on scheme
fs_class = getattr(module, f"{scheme.capitalize()}FileSystem")
# initialize the file system with the kwargs
self.fs = fs_class(**kwargs)
except ImportError:
raise ImportError(
f"Cannot import opendal_service_{scheme}, please check if the module exists"
)
except AttributeError:
raise AttributeError(
f"Cannot find {scheme.capitalize()}FileSystem in opendal_service_{scheme}"
)
```

### Additional Context

_No response_

### Are you willing to contribute to the development of this feature?

- [ ] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

Start with the referenced opendalfs/fs.py example and inspect the existing OpenDAL Python binding package structure. Map the proposed service packages, the virtual opendal meta-package, and on-demand imports; done means the services are split into usable packages while the main package loads the requested service correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
api, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.