pingcap / pingcap/tidb

planner: x BETWEEN y AND y and x >= y AND x <= y cause large performance regression due to missing equi-join extraction

Open
#69,935 2 comments 0 reactions 0 assignees View on GitHub
contribution severity/moderate sig/planner type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

Run the attached reproduction file:

[tidb_range_join_predicate_repro.sql](https://github.com/user-attachments/files/30164089/tidb_range_join_predicate_repro.sql)

[tidb_range_join_predicate_repro_result.txt](https://github.com/user-attachments/files/30164091/tidb_range_join_predicate_repro_result.txt)

The script creates and analyzes three tables containing 6,000 rows each.

The data is constructed so that:

* `t3.c0 = t1.c1` produces 6,000 matching rows.
* `t3.c1 = t2.c1` produces 10 matching rows.
* The final three-table join produces 10 rows.

The script compares the following three logically equivalent queries.

Equality reference:

```sql
SELECT t1.c1 AS ref0, t2.c1 AS ref1
FROM t3
INNER JOIN t1 ON t3.c0 = t1.c1
INNER JOIN t2 ON t3.c1 = t2.c1
WHERE t2.c1 <= 1596112929;
```

`BETWEEN` form:

```sql
SELECT t1.c1 AS ref0, t2.c1 AS ref1
FROM t3
INNER JOIN t1 ON t3.c0 = t1.c1
INNER JOIN t2 ON t3.c1 BETWEEN t2.c1 AND t2.c1
WHERE t2.c1 <= 1596112929;
```

Paired range form:

```sql
SELECT t1.c1 AS ref0, t2.c1 AS ref1
FROM t3
INNER JOIN t1 ON t3.c0 = t1.c1
INNER JOIN t2
ON t3.c1 >= t2.c1
AND t3.c1 <= t2.c1
WHERE t2.c1 <= 1596112929;
```

All three queries return the same result:

```text
+--------------------+-------------+----------+
| variant | result_rows | checksum |
+--------------------+-------------+----------+
| EQUALITY_REFERENCE | 10 | 40000110 |
| BETWEEN_TEST | 10 | 40000110 |
| GE_LE_TEST | 10 | 40000110 |
+--------------------+-------------+----------+
```

The tables were analyzed immediately before the queries were executed:

```text
+------------+--------------+-----------+
| Table_name | Modify_count | Row_count |
+------------+--------------+-----------+
| t1 | 0 | 6000 |
| t2 | 0 | 6000 |
| t3 | 0 | 6000 |
+------------+--------------+-----------+
```

### 2. What did you expect to see? (Required)

TiDB should recognize the following equality-equivalent predicates:

```sql
x BETWEEN y AND y
```

and:

```sql
x >= y AND x <= y
```

as equivalent to:

```sql
x = y
```

The predicates should be normalized before join-condition extraction, allowing the optimizer to construct an equi-join key such as:

```text
equal:[eq(t3.c1, t2.c1)]
```

The resulting physical plan does not have to be identical to the equality reference, but it should avoid a Cartesian join and have comparable performance.

### 3. What did you see instead (Required)

For the equality reference, TiDB correctly extracts both equality predicates as equi-join keys:

```text
HashJoin
inner join,
equal:[eq(t3.c0, t1.c1)]

HashJoin
inner join,
equal:[eq(t2.c1, t3.c1)]
```

The equality reference returned 10 rows with the following execution times:

```text
run 1: 13.4 ms
run 2: 7.92 ms
run 3: 5.65 ms
median: 7.92 ms
```

For the `BETWEEN` form, TiDB expands the predicate into `GE` and `LE` expressions but does not recognize that the lower and upper bounds are identical.

The condition is kept as a residual predicate on a Cartesian Hash Join:

```text
HashJoin
CARTESIAN inner join,
other cond:
ge(t3.c1, t2.c1),
le(t3.c1, t2.c1)
```

The plan estimates that the Cartesian join returns 11,964,000 rows, while the actual output contains only 10 rows:

```text
estRows: 11964000.00
actRows: 10
```

The `BETWEEN` query required:

```text
run 1: 2.26 s
run 2: 1.94 s
run 3: 2.08 s
median: 2.08 s
```

The median runtime is approximately:

```text
2.08 s / 7.92 ms = 262.6x
```

slower than the equality reference.

The explicit paired range form:

```sql
t3.c1 >= t2.c1
AND t3.c1 <= t2.c1
```

produces the same plan shape:

```text
HashJoin
CARTESIAN inner join,
other cond:
ge(t3.c1, t2.c1),
le(t3.c1, t2.c1)
```

This shows that the issue is not specific to the surface syntax of `BETWEEN`. Both forms reach the same internal pair of range conditions, but TiDB does not collapse them into an equality predicate before equi-join extraction.

The other join condition remains a normal non-Cartesian join:

```text
MergeJoin
inner join,
left key:t3.c0,
right key:t1.c1
```

Therefore, the Cartesian join is isolated to the failure to normalize the equality-equivalent range predicate.

#### Suggested fix direction

Before extracting join keys, normalize predicates with identical lower and upper bounds.

For example:

```sql
x BETWEEN y AND y
```

should be normalized to:

```sql
x = y
```

Similarly:

```sql
x >= y AND x <= y
```

should be normalized to:

```sql
x = y
```

The resulting equality expression should participate in equi-join condition extraction so that Hash Join, Merge Join, or Index Join can use it as a join key instead of evaluating it as a residual condition on a Cartesian join.

Regression tests should cover:

* `x BETWEEN y AND y`
* `x >= y AND x <= y`
* NULL values, to verify that SQL three-valued semantics are preserved
* Operand-order variants such as `y <= x AND y >= x`

#### Relation to #69933

This issue is related to #69933 because both cases fail to extract an equality-equivalent predicate as an equi-join key.

However, the missing normalization rules are different.

Issue #69933 concerns negation elimination:

```text
NOT (x <> y) -> x = y
```

This issue concerns collapsing identical lower and upper range bounds:

```text
x BETWEEN y AND y -> x = y

x >= y AND x <= y -> x = y
```

Although both issues have the same downstream symptom—a Cartesian join with a residual condition—they involve distinct predicate rewrite paths and should be tracked separately.

### 4. What is your TiDB version? (Required)

```text
Release Version: v8.5.7
Edition: Community
Git Commit Hash: 202b7f47286a1109b5c957401d34c9358d130ae0
Git Branch: HEAD
UTC Build Time: 2026-07-15 02:06:00
GoVersion: go1.25.10
Race Enabled: false
Check Table Before Drop: false
Store: tikv
```

Relevant optimizer settings:

```text
tidb_opt_enable_hash_join = ON
tidb_hash_join_version = legacy
tidb_cost_model_version = 2
```

Contributor guide

Open the contributing guide

Research direction

Run the attached tidb_range_join_predicate_repro.sql reproduction and compare its equality, BETWEEN, and paired-range plans. Trace predicate normalization and equi-join extraction, then add regression coverage for both forms, operand order, and NULL semantics; done means the equivalent predicates produce usable join keys without a Cartesian join.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.