ClickHouse / ClickHouse/ClickHouse

Sort data at the latest step during formatting

Open
#78,129 0 comments 0 reactions 0 assignees View on GitHub
comp-query-execution external feature minor
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

_No response_

### Use case

Few chunks (20) with small number (20k) of heavy rows (>100KB).
In total it leads to few GB of data.
For those conditions having 3 sorting steps:
1. PartialSortingTransform (parallel)
2. MergeSortingTransform (parallel)
3. MergeSortedTransform (single threaded)

Increase memory usage (due creation of new chunks) and takes a lot of time.

Idea is to make OutputFormat which will collect all chunks (with permutation column potentially) and will read rows in order of permutation and format them in that order.

### Describe the solution you'd like

New OutputFormat transform

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.