apache / apache/datafusion-ballista

Add PartialReduce operator / optimization

Open
#2,314 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Partial-reduce applies a second agreggation to already aggregated results, often significantly reducing the shuffle size for aggregations - significantly reducing shuffle write / read costs.

**Describe the solution you'd like**
Add partial reduce to the stages that further reduces aggregation.
This can be implemented by applying a repartioning + partial reduce in the same stage.
A threshold probably has to be added to avoid slowing down almost-unique inputs.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by tracing the aggregation stages and the repartitioning and shuffle path described in the issue. Determine where a second aggregation can be applied and how a threshold should avoid slowing down nearly unique inputs; done means aggregation stages reduce shuffle data without regressing those inputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.