apache / apache/datafusion

Looser coupling with tokio

Open
#12,784 4 comments 4 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

We would like to use datafusion in a thread-per-core/shared-nothing architecture, for that and other reasons we'd like to not use tokio as the async runtime.

### Describe the solution you'd like

Remove uses of concrete tokio things where not truly necessary, and put truly required uses behind a feature so that users that don't want to use (or even pull in) tokio can still accomplish that.

Based on a shallow analysis, the core and physical plan packages would be the most challenging ones, the others all already don't, or use tokio only as a dev dependency, or the usage is minimal and could be easily removed.

### Describe alternatives you've considered

I don't see a way other than forking datafusion as an alternative to this.

### Additional context

I think it's fine to still treat tokio as the first class citizen, and not provide alternatives within datafusion for cases where it's unavoidable. I imagine anyone who wants this level of access wants extremely high control over I/O abstractions anyway.

Contributor guide

Open the contributing guide

Research direction

Start by auditing concrete Tokio uses in the core and physical plan packages, identified as the most challenging areas. Compare those uses with the other packages, where Tokio is absent, dev-only, or minimal. Done means unnecessary Tokio coupling is removed and unavoidable uses are behind a feature while preserving the supported first-class Tokio path.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.