[FEA] Use `packed_columns` to store results of temporary aggregations in hash-based groupby
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
In hash-based groupby aggregations, in case we have many (hundreds+ to thousands+) aggregations such as `MEAN` or `M2`, we have to create another hundreds+ to thousands+ temporary aggregations such as `COUNT_VALID` and `SUM`. The results of these temporary aggregations are not returned to the users but will be discarded. However, operations such as memory allocation/deallocation and initialization for the results of these temporary aggregations incur a significant amount of overhead.
**Describe the solution you'd like**
We can do better by using `packed_columns` to allocate big chunks of memory to store the results of multiple temporary aggregations. By doing so, we can reduce the number of allocation/deallocation and initialization operations.
Note that if we request a single big chunk of memory that is too big, memory allocation may fail due to fragmentation. As such, if we need too much of temporary memory (such as several GB), instead of requesting a single big chunk of memory, we should request multiple chunks having reasonable size (such as 128MB or 256MB etc).
**Describe alternatives you've considered**
TBA
**Additional context**
TBA.
Contributor guide
Assessment
This issue has not been assessed yet.