ClickHouse / ClickHouse/ClickHouse
Joins use all partitions instead of filtering by partition key
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
When performing a JOIN between two tables where the joined table is partitioned by a specific key, ClickHouse does not filter and only use the relevant partitions based on the join condition. Instead, it scans all partitions, resulting in slower query performance than expected.
**How to reproduce**
Version: 24.2.2.71
```
CREATE TABLE t1 (
id Int,
t2_id Int
)
ENGINE=MergeTree()
PRIMARY KEY id
ORDER BY id;
INSERT INTO t1 (id, t2_id) VALUES
(1, 1),
(2, 2),
(3, 3);
CREATE TABLE t2 (
id Int,
type String
)
ENGINE=MergeTree()
PRIMARY KEY id
PARTITION BY type
ORDER BY id;
INSERT INTO t2 (id, type) VALUES
(1, 'test'),
(2, 'test'),
(3, 'test'),
(4, 'foo');
```
The following query will use all partitions in t2 table:
```
SELECT * FROM t1
LEFT JOIN t2 ON t1.t2_id = t2.id AND t2.type = 'test'
```
```
SELECT partitions
FROM system.query_log
WHERE query_id = 'dbd2af92-cc3d-44bb-b745-7be80d05e007'
┌─partitions────────────────────────────────────────────────────────────────────────────────────────────┐
│ ['test.t1.all','test.t2.20b975a338215f83f5a0a44962ae59ef','test.t2.2a1dedbe81258ce26f3c450185e55508'] │
│ ['test.t1.all','test.t2.20b975a338215f83f5a0a44962ae59ef','test.t2.2a1dedbe81258ce26f3c450185e55508'] │
└───────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
**Expected performance**
The query should only scan partitions of the t2 table where the partition key type matches 'test'. This would significantly reduce the data scanned and thus improve the performance of the query.
Contributor guide
Assessment
This issue has not been assessed yet.