new feature: Support creating and registering custom services via FFI
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Feature Description
Enable third-party applications to register custom OpenDAL services (implementations of Access) across FFI boundaries, so that external Rust crates compiled as separate shared libraries can provide new storage backends without compile-time coupling.
**Desired outcome**
An extension author can:
1. Implement an OpenDAL service in Rust (following the standard Access pattern, like https://github.com/apache/opendal/tree/main/core/services/huggingface)
2. Compile it as a standalone shared library (no compile-time dependency on the host application)
3. Register it with the host application at runtime via some ABI-stable mechanism
4. The host application wraps it as a normal Operator with its own layers (retry, metrics, etc.)
The existing C-ABI types (opendal_bytes, opendal_error, opendal_metadata, etc.) seem like natural building blocks for this.
### Problem and Solution
We're integrating OpenDAL into https://github.com/Eventual-Inc/Daft (a distributed dataframe engine) to support pluggable storage backends. Our users want to implement custom protocols (e.g. `my-proprietary-protocol://`) by writing an OpenDAL service in Rust, packaging it as a Python wheel (cdylib via PyO3/maturin), and registering it at runtime.
The challenge is that Daft and the extension are separate shared libraries loaded into the same Python process. They each statically link their own copy of OpenDAL, so Rust types can't be safely shared across the dylib boundary.
Current state:
OpenDAL's C bindings let you consume services (call opendal_operator_read, etc.), but there's no way to implement a service from the other side of an FFI boundary. The OperatorRegistry requires compile-time generics (register), so you can't register a service from a separately compiled shared library.
### Additional Context
Use cases
- Daft — pluggable storage protocols for a dataframe engine _(the motivating use case)_
- Any application embedding OpenDAL that wants to allow third-party storage plugins without recompilation
- Cross-language service implementations — implement an OpenDAL service in Go, Python (via cffi), etc.
### Are you willing to contribute to the development of this feature?
- [ ] Yes, I am willing to contribute to the development of this feature.
Contributor guide
Research direction
Start with the existing C bindings, including opendal_bytes, opendal_error, opendal_metadata, and the opendal_operator_* entry points, then compare them with the Access pattern in core/services/huggingface and the generic OperatorRegistry registration API. Done should mean a separately compiled service can register across the FFI boundary and be wrapped as a normal Operator with layers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, python, rust
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100