ClickHouse / ClickHouse/ClickHouse
Sort data at the latest step during formatting
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
_No response_
### Use case
Few chunks (20) with small number (20k) of heavy rows (>100KB).
In total it leads to few GB of data.
For those conditions having 3 sorting steps:
1. PartialSortingTransform (parallel)
2. MergeSortingTransform (parallel)
3. MergeSortedTransform (single threaded)
Increase memory usage (due creation of new chunks) and takes a lot of time.
Idea is to make OutputFormat which will collect all chunks (with permutation column potentially) and will read rows in order of permutation and format them in that order.
### Describe the solution you'd like
New OutputFormat transform
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.