apache / apache/datafusion

Manage group values and states by blocks in aggregation

Open
#11,931 11 comments 1 reaction 1 assignee Claimed by @Rachelint View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Now we manage the group values and the aggregation states by a single big vector growing constantly.
This solution is simple to impl, but really leads to some extra cpu cost according to the cpu profile.
Maybe we should manage them by blocks like duckdb.

### Describe the solution you'd like
It may be a big work, I want to finish it through following steps:
- Sketch the total procedure.
- Impl the block based group values management in `GroupValuesRows`.
- Impl the block based `group values` management in other `GroupValues` impls.
- Impl the block based `states` management in different `GroupAccumulator` impls.

The general design is similar as #7065 , but introduce it into GroupValues, not only GroupAccumulators.

### Describe alternatives you've considered

_No response_

### Additional context
The cpu cost flamegraph:
https://github.com/Rachelint/drawio-store/blob/main/cpucosts0811.png

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.