apache / apache/opendal

new feature: Support creating and registering custom services via FFI

Open
#7,194 1 comment 2 reactions 0 assignees View on GitHub
enhancement releases-note/feat
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Enable third-party applications to register custom OpenDAL services (implementations of Access) across FFI boundaries, so that external Rust crates compiled as separate shared libraries can provide new storage backends without compile-time coupling.

**Desired outcome**

An extension author can:
1. Implement an OpenDAL service in Rust (following the standard Access pattern, like https://github.com/apache/opendal/tree/main/core/services/huggingface)
2. Compile it as a standalone shared library (no compile-time dependency on the host application)
3. Register it with the host application at runtime via some ABI-stable mechanism
4. The host application wraps it as a normal Operator with its own layers (retry, metrics, etc.)

The existing C-ABI types (opendal_bytes, opendal_error, opendal_metadata, etc.) seem like natural building blocks for this.

### Problem and Solution

We're integrating OpenDAL into https://github.com/Eventual-Inc/Daft (a distributed dataframe engine) to support pluggable storage backends. Our users want to implement custom protocols (e.g. `my-proprietary-protocol://`) by writing an OpenDAL service in Rust, packaging it as a Python wheel (cdylib via PyO3/maturin), and registering it at runtime.

The challenge is that Daft and the extension are separate shared libraries loaded into the same Python process. They each statically link their own copy of OpenDAL, so Rust types can't be safely shared across the dylib boundary.

Current state:

OpenDAL's C bindings let you consume services (call opendal_operator_read, etc.), but there's no way to implement a service from the other side of an FFI boundary. The OperatorRegistry requires compile-time generics (register), so you can't register a service from a separately compiled shared library.

### Additional Context

Use cases

- Daft — pluggable storage protocols for a dataframe engine _(the motivating use case)_
- Any application embedding OpenDAL that wants to allow third-party storage plugins without recompilation
- Cross-language service implementations — implement an OpenDAL service in Go, Python (via cffi), etc.

### Are you willing to contribute to the development of this feature?

- [ ] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

Start with the existing C bindings, including opendal_bytes, opendal_error, opendal_metadata, and the opendal_operator_* entry points, then compare them with the Access pattern in core/services/huggingface and the generic OperatorRegistry registration API. Done should mean a separately compiled service can register across the FFI boundary and be wrapped as a normal Operator with layers.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python, rust
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.