ClickHouse / ClickHouse/ClickHouse
Optimization limitations of ANTI JOIN with empty table
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Describe the situation
Hi, ClickHouse Developers,
Please consider such two ANTI JOIN queries:
```sql
q1: SELECT * FROM empty_table WHERE empty_table.number NOT IN (SELECT numbers.number FROM numbers);
q2: SELECT * FROM empty_table WHERE NOT EXISTS (SELECT 1 FROM numbers);
```
These two ANTI JOIN queries are semantically equivalent. When you execute these two queries, the first query(q1) will process the complete numbers table, while the second one will quickly return empty results. Since the second query(q2) quickly returns an empty set of results, I think that CIickHouse has made optimizations for empty tables. So the first query(q2) indicates a performance bug.
Obviously, the above query always returns an empty set. I think it should return an empty set quickly. I think this is a common case in actual production scenarios. It's important to clarify that users **might not intentionally perform ANTI JOIN operations on empty tables**. Rather, **they may be unaware that a table is empty.** For example, when data has been deleted by another user or process.
### Which ClickHouse versions are affected?
I ran this case on the last two versions: 25.4.2 and 25.5.1
### How to reproduce
We can reproduce it as follow:
```sql
e8fe431007a5 :) SET default_table_engine = 'Log';
e8fe431007a5 :) CREATE TABLE empty_table AS SELECT * FROM numbers LIMIT 0;
e8fe431007a5 :) SELECT * FROM empty_table WHERE empty_table.number NOT IN (SELECT * FROM numbers);
SELECT *
FROM empty_table
WHERE empty_table.number NOT IN (
SELECT numbers.number
FROM numbers
)
Query id: 97231c21-5f3b-4645-a896-ee8a865a72b4
Cancelling query.
Query was cancelled.
0 rows in set. Elapsed: 10.644 sec. Processed 134.28 million rows, 1.07 GB (12.62 million rows/s., 100.92 MB/s.)
Peak memory usage: 5.99 GiB.
e8fe431007a5 :) SELECT * FROM empty_table WHERE NOT EXISTS (SELECT 1 FROM numbers);
SELECT *
FROM empty_table
WHERE NOT exists((
SELECT 1
FROM numbers
))
Query id: 4b1ac6ed-f909-4064-9fb0-8924999f8773
Ok.
0 rows in set. Elapsed: 0.022 sec.
e8fe431007a5 :) EXPLAIN SELECT * FROM empty_table WHERE empty_table.number NOT IN (SELECT numbers.number FROM numbers);
EXPLAIN
SELECT *
FROM empty_table
WHERE empty_table.number NOT IN (
SELECT numbers.number
FROM numbers
)
Query id: 6711f463-4d70-4b56-b0d3-b9179d66319c
┌─explain─────────────────────────────────────────────────────────────────────────────────────┐
1. │ CreatingSets (Create sets before main query execution) │
2. │ Expression ((Project names + Projection)) │
3. │ Filter ((WHERE + Change column names to column identifiers)) │
4. │ ReadFromStorage (Log) │
5. │ CreatingSet (Create set for subquery) │
6. │ Expression ((Project names + (Projection + Change column names to column identifiers))) │
7. │ ReadFromSystemNumbers │
└─────────────────────────────────────────────────────────────────────────────────────────────┘
7 rows in set. Elapsed: 0.006 sec.
e8fe431007a5 :) EXPLAIN SELECT * FROM empty_table WHERE NOT EXISTS (SELECT 1 FROM numbers);
EXPLAIN
SELECT *
FROM empty_table
WHERE NOT exists((
SELECT 1
FROM numbers
))
Query id: ef160a69-fc6f-4a1f-b347-089429b8ed1b
┌─explain────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
1. │ CreatingSets (Create sets before main query execution) │
2. │ Expression ((Project names + Projection)) │
3. │ Filter ((WHERE + Change column names to column identifiers)) │
4. │ ReadFromStorage (Log) │
5. │ CreatingSet (Create set for subquery) │
6. │ Expression ((Project names + (Projection + (Change column names to column identifiers + (Project names + (Projection + Change column names to column identifiers)))))) │
7. │ Limit (preliminary LIMIT (without OFFSET)) │
8. │ ReadFromSystemNumbers │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
8 rows in set. Elapsed: 0.016 sec.
```
### Expected performance
The q1 returns an empty set quickly as follow:
```sql
e8fe431007a5 :) SELECT * FROM empty_table WHERE empty_table.number IN (SELECT numbers.number FROM numbers);
...
0 rows in set. Elapsed: 0.xxx sec.
```
Contributor guide
Assessment
This issue has not been assessed yet.