apache / apache/opendal

new feature: compio executor support

Open
#8,037 0 comments 5 reactions 0 assignees View on GitHub
core enhancement releases-note/feat
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Feature Description

Add a new `Executor` under core that uses compio (compio-dispatcher, to be particular) to drive a future.

### Problem and Solution

There are 2 major obstacles:
- OpenDAL is dominently multithreaded, while compio is single threaded, meaning lots of the future returned by the runtime (include timeout) is `!Send + !Sync`, which is incompatible with lots of the traits OpenDAL have right now. This is in fact already solved in `service-compfs`, where `compio-dispatcher` is used to send an async task to be executed in a thread pool, so that caller will get a channel for returned value, hence `Send` and `Sync`, thread safe.
- `timeout` layer, with several other places where timeout is needed, currently assume tokio is used, and call `tokio::time::timeout/sleep` directly. This should be changed to a top-down invocation where all timeout/sleep call should be delegated all the way down to `Execute::timeout`. However the current `Executor` design makes this quite impossible. A change to `Executor`'s `timeout` function is needed.

### Additional Context

_No response_

### Are you willing to contribute to the development of this feature?

- [x] Yes, I am willing to contribute to the development of this feature.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing Executor design under core and the compio-dispatcher usage in service-compfs. Trace the timeout layer and the Execute::timeout entry point to understand where Tokio is assumed. Done means a compio-backed Executor can drive futures safely and timeout/sleep calls are delegated through the executor design.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.