apache / apache/datafusion

StateBackend in DataFusion's RuntimeEnv

Open
#11,365 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Currently DataFusion operators communicate via a narrow API i.e. forwarding `SendableRecordBatchStreams`. In some instances, in particular the ExecutionPlans operating on unbounded streams need to snapshot their state and co-ordinate with source operators. It'd be a powerful primitive to add a `StateBackend` concept to the `RuntimeEnv` where users could then write operators to store adhoc durable state into a backend such as rocksdb.

Realise this may not be useful for many use cases but [RuntimeEnv](https://github.com/apache/datafusion/blob/e65c3e919855c9977cf4d80c0630ee26b7fd03ee/datafusion/execution/src/runtime_env.rs#L46-L55) does seem to have ability to plug in an object store registry as well as a catalog manager. This would be a crucial unlock to make stateful stream processing application with DataFusion.

If the current, API contains such a pathway already, would love to get pointers in the right direction.

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/execution/src/runtime_env.rs, especially RuntimeEnv and its object store registry and catalog manager integrations. Trace whether an existing pathway supports durable state for unbounded ExecutionPlans; done requires a maintainer-approved StateBackend design, with its intended coordination and persistence behavior made explicit.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.