apache / apache/opendal

(bindings/python)Split OpenDAL python bindings into separate packages

Open
#6,748 6 comments 2 reactions 0 assignees View on GitHub
bindings/python enhancement releases-note/feat
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

I have a goal to finally split OpenDAL into separate crates so users can easily implement and use their own services. On the Python side, I want to dynamically load them. Is there a way to enable this?

This is not a blocker: we can still merge this PR as is, but let’s keep this idea in mind as we move forward.

_Originally posted by @Xuanwo in https://github.com/apache/opendal/pull/6729#discussion_r2456026266_

---

## Approach
I've found an approach for this which is semi automated and requires some templating magic, but it should work theoretically.

I tried it in a `uv` workspace with python-only workspace members and it worked. need to now test this with `maturin` since `opendal` workspace members are rust+python (`pyo3`).

heavily inspired from : https://github.com/santosh-d3vpl3x/duckdb_extensions

### Goals
- Different services have different levels of maintenance and testing levels which can be highlighted
- Different maturity levels can be added to services as well potentially.

> Like s3, fs, … are stable, welcome to use. ftp had tests covered but no known users, use carefully
- Reduction in size of base wheel

### Non Goals

- There is a potential to make the python api look similar to that of fsspec alongwith all of the other changes. At the end, this will just be from interface side and can make opendal directly be a `drop-in` replacement, where ever a `fsspec` `AsyncFileSystem` is needed.

## What this will look like?

just like the extensions dir in the [above repo](https://github.com/santosh-d3vpl3x/duckdb_extensions), I'm planning for 2 dirs to start with - services and layers. These represent a class of workspace members.

opendal's uv workspace will have these as members. root package is not published as its an aggregator, but the members are.

### Example of members
- opendal-service-hdfs
- opendal-service-sftp ... and so on
- opendal-layer-tailcut
- opendal-layer-otel-metrics ... and so on
- opendal (lets call this `base` for now. this is equivalent to the existing package with all the core like stuff and maybe some default services and layers)

### Defaults in opendal package

To keep things minimal in the opendal base package, we can ship it,
- service-fs
- service-memory
- service-s3
- service-gcs
- service-azure
- layer-retry
- layer-concurrent-limit
- layer-mime-guess

This covers the existing 3 layers and all the major cloud providers.

## Considerations
- opendal's python bindings before and after this refactoring will not be compatible with each other as current implementation is monolithic in 1 wheel and the future ones are split into packages
- Since the packages are split, their tests, documentation, API references, stubs etc. will be separate as well. So need to think about aggregation, deployments and executions.
- Since major breaking changes are introduced, need to think about versioning this.
- current overloads of Operator and AsyncOperator (`__new__(...)`) will be useless and will have to be removed.
- more.... ?

## Additional Context
In line with `Plan A`: https://github.com/apache/opendal/discussions/5206

---

Have a look when you can.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the proposed uv workspace structure, the duckdb_extensions example, and the linked Plan A discussion. Test the approach with maturin for the Rust and Python workspace, then define how the split packages, tests, documentation, API references, stubs, deployments, and versioning will be aggregated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
build-system, developer-experience
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.