Speed up hash partitioning
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Also see request in arrow https://github.com/apache/arrow-rs/issues/4476
In DataFusion, a common operation is to repartition a RecordBatch by hashing one or more columns and dividing them into partition record batches using the "formula" hash % num_partitions.
The current approach is to create the indices that match and use them to take the individual arrays (see BatchPartitioner in datafusion).
This is relatively expensive however, as we visit the arrays num_partitions times in different places of the array, leading to cache inefficient operators (especially when the number of partitions is high).
### Describe the solution you'd like
Faster hash-partitioning implementation
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reading the BatchPartitioner implementation in DataFusion and the linked Arrow Rust request to understand the current index-and-take approach. Measure hash repartitioning with different numbers of partitions, then compare a faster implementation against the existing behavior and benchmarks; done means equivalent partitioned RecordBatches with improved performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100