apache / apache/datafusion-ballista

Design: Implement bubble execution mode.

Open
#408 2 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Currently ballista uses schedules stage by stage, which is great for fault tolerance, but inefficient for adhoc olap execution. Bubble execution mode is to make ballista more efficient for adhoc olap queries which doesn't require fault tolerance.

**Describe the solution you'd like**
The detailed design can be found [here](https://docs.google.com/document/d/1OdAe078axk4qO0ozUxNqBMD4wKoBhzh9keMuLp_jerE/edit?usp=sharing), welcome to comment and share your thoughts.

**Describe alternatives you've considered**
No.

**Additional context**
No.

Contributor guide

Open the contributing guide

Research direction

Start with the linked detailed design document and compare it with Ballista's current stage-by-stage scheduling approach. The issue names no files or tests; completion would mean implementing the proposed bubble execution mode for ad hoc OLAP queries while preserving the stated trade-off of not requiring fault tolerance.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.