apache / apache/datafusion

Add support for Range partitioning

Open
#3,628 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In order to support parallel sort (or distributed sort in Ballista) we need to implement range partitioning.

**Describe the solution you'd like**
We should support partitioning by range.
A range is a vector of buckets (for example `[0, 100, 200, 400, 600, 1000]`) combined with a expression.
The range vector might be given or computed/updated during query execution.

Each value belongs to partition i when the value is between the range values i and i + 1.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing partitioning implementation and the parallel-sort or distributed-sort path in DataFusion. Review how partition expressions and partition counts are represented, then define the range-bucket behavior, including runtime-computed or updated boundaries. Done means values are assigned to the correct range partitions and the behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.