planner: contradictory IS NULL and derived IS NOT NULL predicates are not folded to TableDual
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce step (Required)
[tidb_impossible_null_equality_repro.sql](https://github.com/user-attachments/files/30160120/tidb_impossible_null_equality_repro.sql)
[tidb_impossible_null_equality_repro_result.txt](https://github.com/user-attachments/files/30160122/tidb_impossible_null_equality_repro_result.txt)
The optimizer does not fold contradictory `IS NULL` and `IS NOT NULL`
predicates into an empty result. The same problem also occurs when
`IS NOT NULL` is derived from an ordinary equality join.
The following test creates 100,000 non-NULL rows and 500,000 NULL rows
in each table.
```sql
DROP DATABASE IF EXISTS tidb_null_contradiction_repro;
CREATE DATABASE tidb_null_contradiction_repro;
USE tidb_null_contradiction_repro;
CREATE TABLE digits (
n INT NOT NULL PRIMARY KEY
);
INSERT INTO digits VALUES
(0), (1), (2), (3), (4),
(5), (6), (7), (8), (9);
CREATE TABLE seq (
n BIGINT NOT NULL PRIMARY KEY
);
INSERT INTO seq(n)
SELECT
d0.n
+ 10 * d1.n
+ 100 * d2.n
+ 1000 * d3.n
+ 10000 * d4.n
+ 100000 * d5.n
FROM digits AS d0
CROSS JOIN digits AS d1
CROSS JOIN digits AS d2
CROSS JOIN digits AS d3
CROSS JOIN digits AS d4
CROSS JOIN digits AS d5
WHERE d5.n < 5;
CREATE TABLE t0 (
id BIGINT NOT NULL PRIMARY KEY,
c0 BIGINT NULL,
c3 BIGINT NOT NULL,
KEY i0 (c0)
);
CREATE TABLE t1 (
id BIGINT NOT NULL PRIMARY KEY,
c0 BIGINT NULL,
c1 BIGINT NOT NULL,
KEY i1 (c0, c1)
);
-- 100,000 matching non-NULL rows.
INSERT INTO t0
SELECT n, n, MOD(n, 1024)
FROM seq
WHERE n < 100000;
INSERT INTO t1
SELECT n, n, MOD(n, 2048)
FROM seq
WHERE n < 100000;
-- 500,000 NULL rows.
INSERT INTO t0
SELECT 100000 + n, NULL, MOD(n, 1024)
FROM seq;
INSERT INTO t1
SELECT 100000 + n, NULL, MOD(n, 2048)
FROM seq;
ANALYZE TABLE t0;
ANALYZE TABLE t1;
SELECT
COUNT(*) AS t0_rows,
SUM(c0 IS NULL) AS t0_null_rows
FROM t0;
SELECT
COUNT(*) AS t1_rows,
SUM(c0 IS NULL) AS t1_null_rows
FROM t1;
```
The data shape is:
```text
t0_rows = 600000, t0_null_rows = 500000
t1_rows = 600000, t1_null_rows = 500000
```
First, TiDB can correctly produce `TableDual` for an explicit false
condition:
```sql
EXPLAIN
SELECT t1.c1, t0.c0
FROM t1
JOIN t0 ON t1.c0 = t0.c0
WHERE 1 = 0;
```
The plan is:
```text
TableDual
rows:0
```
However, a direct contradiction on one nullable column is not folded:
```sql
EXPLAIN
SELECT c1
FROM t1
WHERE c0 IS NULL
AND c0 IS NOT NULL;
```
The generated plan contains:
```text
IndexReader
└─Selection not(isnull(t1.c0))
└─IndexRangeScan range:[NULL,NULL]
```
The following ordinary inner equality join is also logically
impossible:
```sql
EXPLAIN ANALYZE
SELECT t1.c1 AS ref0, t0.c0
FROM t1
JOIN t0 ON t1.c0 = t0.c0
WHERE t1.c0 IS NULL
ORDER BY t0.c3, t0.c0 DESC
LIMIT 2;
```
Because ordinary equality is NULL-rejecting, `t1.c0 IS NULL` and
`t1.c0 = t0.c0` can never both evaluate to TRUE.
The query correctly returns zero rows, but its plan contains:
```text
IndexRangeScan range:[NULL,NULL]
└─Selection not(isnull(t1.c0))
```
The actual execution information is:
```text
execution time: 292.6 ms
RU: 448.60
total_process_keys: 500000
IndexRangeScan rows: 500000
Selection rows: 0
final result rows: 0
```
A symmetric query provides even more direct evidence:
```sql
EXPLAIN ANALYZE
SELECT t1.c1 AS ref0, t0.c0
FROM t1
JOIN t0 ON t1.c0 = t0.c0
WHERE t0.c0 IS NULL
ORDER BY t0.c3, t0.c0 DESC
LIMIT 2;
```
The generated `Selection` explicitly contains both mutually exclusive
predicates:
```text
isnull(t0.c0), not(isnull(t0.c0))
```
Nevertheless, TiDB still executes the plan:
```text
TableFullScan t0 rows: 600000
IndexFullScan t1 rows: 100000
Selection rows: 0
execution time: 385.6 ms
RU: 684.92
HashJoin memory: 5.40 MB
```
The issue does not depend on `ORDER BY` or `LIMIT`. The following query
also produces a join and NULL-range scan instead of an empty plan:
```sql
EXPLAIN
SELECT t1.c1, t0.c0
FROM t1
JOIN t0 ON t1.c0 = t0.c0
WHERE t1.c0 IS NULL;
```
For semantic verification:
```sql
SELECT COUNT(*) AS impossible_join_rows
FROM t1
JOIN t0 ON t1.c0 = t0.c0
WHERE t1.c0 IS NULL;
SELECT EXISTS (
SELECT 1
FROM t1
JOIN t0 ON t1.c0 <=> t0.c0
WHERE t1.c0 IS NULL
) AS null_safe_join_has_rows;
SELECT EXISTS (
SELECT 1
FROM t1
JOIN t0 ON t1.c0 = t0.c0
WHERE t1.c0 IS NOT NULL
) AS ordinary_nonnull_join_has_rows;
```
The results are:
```text
impossible_join_rows = 0
null_safe_join_has_rows = 1
ordinary_nonnull_join_has_rows = 1
```
Therefore, the zero-row result is caused by SQL NULL semantics rather
than by the absence of matching data.
### 2. What did you expect to see? (Required)
TiDB should detect the contradictory predicates during logical
optimization.
For the direct case:
```sql
c0 IS NULL AND c0 IS NOT NULL
```
the predicate is always false.
For the join case:
```sql
t1.c0 IS NULL
AND t1.c0 = t0.c0
```
ordinary equality implies that both equality operands must be non-NULL.
TiDB already derives and displays `not(isnull(t1.c0))` in the plan.
This derived predicate contradicts the original `t1.c0 IS NULL`
predicate.
The optimizer should therefore replace the query subtree with
`TableDual`, or another zero-row plan that does not access storage:
```text
TableDual
rows:0
```
The estimated cardinality should also be exactly zero.
### 3. What did you see instead? (Required)
TiDB recognizes part of the contradiction but does not eliminate it.
For example, the same branch contains:
```text
IndexRangeScan range:[NULL,NULL]
Selection not(isnull(t1.c0))
```
Another plan explicitly contains:
```text
isnull(t0.c0), not(isnull(t0.c0))
```
Despite these contradictory predicates, TiDB continues physical
optimization and execution.
With 500,000 NULL rows, a default plan without hints:
- reads 500,000 NULL index entries;
- spends 292.6 ms and 448.60 RU;
- filters every scanned row;
- returns zero rows.
The symmetric form performs more unnecessary work:
- scans all 600,000 rows of `t0`;
- reads 100,000 rows from `t1`;
- builds a hash join;
- spends 385.6 ms and 684.92 RU;
- returns zero rows.
This is not a result-correctness issue. The returned result is correct,
but the query performs storage scans and join execution that should be
eliminated during optimization.
#### Suggested fix direction
One possible fix direction is to run contradiction detection after
predicate propagation and NULL-rejecting inference.
When the optimizer has both of the following predicates for the same
expression:
```text
isnull(column)
not(isnull(column))
```
the corresponding logical subtree can be replaced with a zero-row
logical plan before physical plan enumeration.
The same check should apply when `IS NOT NULL` is derived from an
ordinary equality condition rather than written explicitly by the
user.
### 4. What is your TiDB version? (Required)
```text
Release Version: v8.5.7
Edition: Community
Git Commit Hash: unavailable in the current client output
TiDB Version: 8.0.11-TiDB-v8.5.7
```
Output of:
```sql
SELECT tidb_version();
```
```text
8.0.11-TiDB-v8.5.7
```
Contributor guide
Research direction
No source files or tests are named. Start by reproducing the direct contradiction with EXPLAIN and then the equality-join case, focusing on logical optimization after predicate propagation and NULL-rejecting inference. Done means contradictory predicates produce a TableDual or equivalent zero-row plan without storage scans, with estimated cardinality zero.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100