new feature: Split python binding in seperate packages
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
The OpenDAL Python binding now releases all its services in a single package, which makes it difficult to use and extend. I propose splitting it into separate packages, similar to what I've done with [`opendalfs`](https://github.com/fsspec/opendalfs).
```mermaid
graph TD;
opendalfs.OpendalFileSystem -- import --> MemoryFileSystem;
opendalfs.OpendalFileSystem -- import --> S3FileSystem;
opendalfs.OpendalFileSystem -- import --> FsFileSystem;
opendalfs.OpendalFileSystem -- import --> ...FileSystem;
MemoryFileSystem -- use --> opendalfs-core;
S3FileSystem -- use --> opendalfs-core;
FsFileSystem -- use --> opendalfs-core;
...FileSystem -- use --> opendalfs-core;
opendalfs-core -- use -->opendal["Apache OpenDAL"];
```
### Problem and Solution
OpenDAL Python is large yet still doesn't cover all the services users need.
We can divide it into multiple packages, making `opendal` a virtual meta-package that only provides the Python API and imports the correct service on demand when needed.
For example, as I showed up in `opendalfs`:
https://github.com/fsspec/opendalfs/blob/e19d28eb9f82e285685f91b3c80805146759b7d7/opendalfs/fs.py#L8-L25
```python
def __init__(self, scheme, *args, **kwargs):
super().__init__(*args, **kwargs)
try:
# Load the module dynamically based on scheme
module = importlib.import_module(f"opendalfs_service_{scheme}")
# Get the file system class based on scheme
fs_class = getattr(module, f"{scheme.capitalize()}FileSystem")
# initialize the file system with the kwargs
self.fs = fs_class(**kwargs)
except ImportError:
raise ImportError(
f"Cannot import opendal_service_{scheme}, please check if the module exists"
)
except AttributeError:
raise AttributeError(
f"Cannot find {scheme.capitalize()}FileSystem in opendal_service_{scheme}"
)
```
### Additional Context
_No response_
### Are you willing to contribute to the development of this feature?
- [ ] Yes, I am willing to contribute to the development of this feature.
Contributor guide
Research direction
Start with the referenced opendalfs/fs.py example and inspect the existing OpenDAL Python binding package structure. Map the proposed service packages, the virtual opendal meta-package, and on-demand imports; done means the services are split into usable packages while the main package loads the requested service correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- api, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100