ClickHouse / ClickHouse/ClickHouse

optimize_read_in_order degrades to multi-part k‑way merge when PARTITION BY is used, causing slower ORDER BY … LIMIT N

Open
#89,638 1 comment 4 reactions 0 assignees View on GitHub
comp-query-optimizer external unfinished code
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Describe the unexpected behaviour

When a table is partitioned (e.g., PARTITION BY toYYYYMM(dt)) and contains many parts per partition, enabling optimize_read_in_order for a query like SELECT … ORDER BY dt LIMIT 20 produces a pipeline with one reader per part and deep MergingSortedTransform stacks. In practice this is slower than reading in parallel with partial sort. The behavior appears to be a limitation of the implementation rather than a user error, but it’s not documented in the public docs.

* Planner selects `ReadPoolInOrder` with **N = number of parts** streams, followed by multi‑level `MergingSortedTransform`.
* Wall‑clock latency is worse than with `optimize_read_in_order = 0`, even for tiny LIMIT.
* Rows read are lower in the counter, but total query time and memory increase due to the k‑way merge and per‑part readers.

### How to reproduce

https://fiddle.clickhouse.com/53adc384-56ad-4dd9-8afa-b0baa8d154be

```
CREATE TABLE test (dt DateTime,)
ENGINE = MergeTree
PARTITION BY toYYYYMM(dt)
ORDER BY (dt);

set max_partitions_per_insert_block=0;
INSERT INTO test
SELECT toDate('2015-01-01') + (number % 3650) AS dt
FROM numbers(10000000);

SELECT dt FROM test ORDER BY dt LIMIT 20
```

With PARTITION BY commended/removed, the pipeline and query execution time/rows become expected.

### Expected behavior

For `ORDER BY` matching the sorting key and small `LIMIT`, `optimize_read_in_order` should:

* Read only the necessary ranges, minimizing work, and
* Avoid excessive parallel streams proportional to number of parts, or at least short‑circuit early once LIMIT is satisfied, and
* Prioritize reading from most promising partitions (e.g., newest) first.

Alternatively, if the above is not feasible today, the documentation should clearly state the limitations with partitioned tables and high part counts and recommend when to disable the optimization.

### Error message and/or stacktrace

```
(Expression)
ExpressionTransform
(Limit)
Limit
(Sorting)
MergingSortedTransform 8 → 1
(Expression)
ExpressionTransform × 8
(ReadFromMergeTree)
ExpressionTransform × 8
MergingSortedTransform 75 → 1
ExpressionTransform × 75
MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder) × 75 0 → 1
MergingSortedTransform 75 → 1
ExpressionTransform × 75
MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder) × 75 0 → 1
MergingSortedTransform 75 → 1
ExpressionTransform × 75
MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder) × 75 0 → 1
MergingSortedTransform 75 → 1
ExpressionTransform × 75
MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder) × 75 0 → 1
MergingSortedTransform 75 → 1
ExpressionTransform × 75
MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder) × 75 0 → 1
MergingSortedTransform 75 → 1
ExpressionTransform × 75
MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder) × 75 0 → 1
MergingSortedTransform 75 → 1
ExpressionTransform × 75
MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder) × 75 0 → 1
MergingSortedTransform 75 → 1
ExpressionTransform × 75
MergeTreeSelect(pool: ReadPoolInOrder, algorithm: InOrder) × 75 0 → 1

```

### Additional context

* Related open GitHub issue on memory/stream scaling with `optimize_read_in_order` ([https://github.com/ClickHouse/ClickHouse/issues/52624](https://github.com/ClickHouse/ClickHouse/issues/52624))

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.