planner: NOT (x <> y) is not normalized to an equi-join key, causing a Cartesian join and about 247x slowdown
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal reproduce step (Required)
Run the attached reproduction script:
[tidb_not_ne_join_predicate_repro.sql](https://github.com/user-attachments/files/30163947/tidb_not_ne_join_predicate_repro.sql)
[tidb_not_ne_join_predicate_repro_result.txt](https://github.com/user-attachments/files/30163950/tidb_not_ne_join_predicate_repro_result.txt)
The script creates and analyzes three tables with 6,000 rows each.
The data is constructed so that:
* `t3.c0 = t1.c1` produces 6,000 rows.
* `t3.c1 = t2.c1` produces 10 rows.
* The final three-table join produces 10 rows.
It then compares the following two logically equivalent queries.
Equality reference:
```sql
SELECT t1.c1 AS ref0, t2.c1 AS ref1
FROM t3
INNER JOIN t1 ON t3.c0 = t1.c1
INNER JOIN t2 ON t3.c1 = t2.c1
WHERE t2.c1 <= 1596112929;
```
`NOT (<>)` form:
```sql
SELECT t1.c1 AS ref0, t2.c1 AS ref1
FROM t3
INNER JOIN t1 ON t3.c0 = t1.c1
INNER JOIN t2 ON NOT (t3.c1 <> t2.c1)
WHERE t2.c1 <= 1596112929;
```
The two join predicates are equivalent under SQL three-valued logic:
```sql
NOT (x <> y)
```
and:
```sql
x = y
```
Both queries return the same result:
```text
+--------------------+-------------+----------+
| variant | result_rows | checksum |
+--------------------+-------------+----------+
| EQUALITY_REFERENCE | 10 | 40000110 |
| NOT_NE_TEST | 10 | 40000110 |
+--------------------+-------------+----------+
```
All three tables are analyzed before the queries are executed:
```text
+------------+--------------+-----------+
| Table_name | Modify_count | Row_count |
+------------+--------------+-----------+
| t1 | 0 | 6000 |
| t2 | 0 | 6000 |
| t3 | 0 | 6000 |
+------------+--------------+-----------+
```
### 2. What did you expect to see? (Required)
TiDB should recognize:
```sql
NOT (t3.c1 <> t2.c1)
```
as an equality-equivalent join predicate and extract it as an equi-join key, in the same way as:
```sql
t3.c1 = t2.c1
```
The resulting plan does not have to be identical, but it should avoid a Cartesian join and should have performance comparable to the equality form.
For example, the condition should be represented as an equality condition similar to:
```text
equal:[eq(tidb_not_ne_join_repro.t2.c1,
tidb_not_ne_join_repro.t3.c1)]
```
### 3. What did you see instead (Required)
For the equality reference, TiDB correctly extracts both equality predicates as equi-join keys:
```text
HashJoin
inner join,
equal:[eq(tidb_not_ne_join_repro.t3.c0,
tidb_not_ne_join_repro.t1.c1)]
HashJoin
inner join,
equal:[eq(tidb_not_ne_join_repro.t2.c1,
tidb_not_ne_join_repro.t3.c1)]
```
The equality reference returned 10 rows with the following execution times:
```text
run 1: 13.1 ms
run 2: 7.5 ms
run 3: 7.3 ms
median: 7.5 ms
```
However, TiDB does not extract the `NOT (<>)` predicate as an equi-join key.
Instead, it produces a Cartesian Hash Join and evaluates the predicate as a residual condition:
```text
HashJoin
CARTESIAN inner join,
other cond:not(ne(tidb_not_ne_join_repro.t3.c1,
tidb_not_ne_join_repro.t2.c1))
```
The unchanged equality join between `t3` and `t1` remains a normal non-Cartesian join:
```text
MergeJoin
inner join,
left key:tidb_not_ne_join_repro.t3.c0,
right key:tidb_not_ne_join_repro.t1.c1
```
This isolates the Cartesian join to the failure to normalize:
```sql
NOT (t3.c1 <> t2.c1)
```
The Cartesian Hash Join is estimated to return 11,964,000 rows, although the actual result contains only 10 rows:
```text
estRows: 11964000.00
actRows: 10
```
The `NOT (<>)` query returned the same result but required:
```text
run 1: 2.06 s
run 2: 1.85 s
run 3: 1.83 s
median: 1.85 s
```
The median runtime is therefore approximately:
```text
1.85 s / 7.5 ms = 246.7x
```
slower than the logically equivalent equality form.
This appears to be a missing predicate-normalization rule rather than a join-algorithm selection problem. Because `NOT (x <> y)` is not converted into an equality expression before join-condition extraction, TiDB treats it as a general residual condition and cannot construct an equi-join key.
#### Suggested fix direction
Normalize equality-equivalent expressions of the form:
```sql
NOT (x <> y)
```
into:
```sql
x = y
```
before extracting join equality conditions.
The normalized expression should be added to the join's equality-condition list so that Hash Join, Merge Join, or Index Join can use it as a join key instead of executing a Cartesian join with a residual filter.
A regression test should cover both non-NULL and NULL inputs to verify that the rewrite preserves SQL three-valued semantics.
### 4. What is your TiDB version? (Required)
```text
Release Version: v8.5.7
Edition: Community
Git Commit Hash: 202b7f47286a1109b5c957401d34c9358d130ae0
Git Branch: HEAD
UTC Build Time: 2026-07-15 02:06:00
GoVersion: go1.25.10
Race Enabled: false
Check Table Before Drop: false
Store: tikv
```
Relevant optimizer settings:
```text
tidb_opt_enable_hash_join = ON
tidb_hash_join_version = legacy
tidb_cost_model_version = 2
```
Contributor guide
Research direction
Run the attached tidb_not_ne_join_predicate_repro.sql script and compare the equality and NOT (x <> y) plans and timings. Trace the planner's predicate-normalization and join-condition extraction entry points, then add a regression test covering non-NULL and NULL inputs; done means the predicate becomes an equi-join key without changing SQL three-valued semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100