pingcap / pingcap/tidb

TiDB does not simplify `NOT NOT` around an `IN` subquery, causing a large performance regression

Open
#69,926 2 comments 0 reactions 0 assignees View on GitHub
contribution severity/moderate sig/planner type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!
The two queries below are logically equivalent. The mutated query only wraps the `IN` predicate with a double negation:

```sql
x IN (subquery)
NOT NOT x IN (subquery)
```

TiDB v8.5.7 generates very different plans. The original query is optimized into an inner/semi-join style plan with null-rejecting predicates. The double-negated version keeps a `Selection` with `not(not(Column#...))` above a `left outer semi join`, which is much slower.

Observed timing with the minimized SQL over 8 alternating executions:

```text
original median = 2.950 ms
mutated median = 80.244 ms
ratio = 27.20x slower
```

The original round37 query was also rerun and reproduced an even larger difference:

```text
original median = 12.586 ms
mutated median = 543.321 ms
ratio = 43.17x slower
```

Both versions return the same result.

### 1. Minimal reproduce step (Required)

```sql
DROP DATABASE IF EXISTS tidb_notnot_in_min;
CREATE DATABASE tidb_notnot_in_min;
USE tidb_notnot_in_min;

CREATE TABLE a(x INT);
CREATE TABLE b(y INT);

CREATE TABLE d(n INT);
INSERT INTO d VALUES (0),(1),(2),(3),(4),(5),(6),(7),(8),(9);

-- 2,000 rows
INSERT INTO a
SELECT x.n * 1000 + y.n * 100 + z.n * 10 + w.n
FROM d AS x
JOIN d AS y
JOIN d AS z
JOIN d AS w
WHERE x.n < 2;

-- 1,000 rows
INSERT INTO b
SELECT x.n * 1000 + y.n * 100 + z.n * 10 + w.n
FROM d AS x
JOIN d AS y
JOIN d AS z
JOIN d AS w
WHERE x.n < 1;

-- Original query
SELECT COUNT(*)
FROM a
WHERE x IN (SELECT y FROM b WHERE y >= 0);

-- Mutated query: logically equivalent double negation
SELECT COUNT(*)
FROM a
WHERE NOT NOT x IN (SELECT y FROM b WHERE y >= 0);

-- Optional: inspect the plan difference
EXPLAIN
SELECT COUNT(*)
FROM a
WHERE x IN (SELECT y FROM b WHERE y >= 0);

EXPLAIN
SELECT COUNT(*)
FROM a
WHERE NOT NOT x IN (SELECT y FROM b WHERE y >= 0);
```

### 2. What did you expect to see? (Required)
TiDB should simplify `NOT NOT predicate` to `predicate` before subquery optimization, or otherwise apply the same semi-join and null-rejection optimizations to both equivalent predicates. The two queries should have similar execution plans and similar execution time.

```text
COUNT(*) = 1000
```

### 3. What did you see instead (Required)
Original query:

```text
HashAgg_13 funcs:count(1)->Column#5
HashJoin_16 inner join, equal:[eq(b.y, a.x)]
HashAgg_22(Build) group by:b.y
Selection_21 ge(b.y, 0), not(isnull(b.y))
TableFullScan_20 table:b
Selection_28 not(isnull(a.x))
TableFullScan_27 table:a
```

Query with `NOT NOT`:

```text
HashAgg_10 funcs:count(1)->Column#6
Selection_12 not(not(Column#5))
HashJoin_13 CARTESIAN left outer semi join, other cond:eq(a.x, b.y)
Selection_17 ge(b.y, 0)
TableFullScan_16 table:b
TableFullScan_14 table:a
```

### 4. What is your TiDB version? (Required)

```text
Release Version: v8.5.7
Edition: Community
Git Commit Hash: 202b7f47286a1109b5c957401d34c9358d130ae0
Git Branch: HEAD
UTC Build Time: 2026-07-15 02:06:00
GoVersion: go1.25.10
Race Enabled: false
Check Table Before Drop: false
Store: unistore
```

Contributor guide

Open the contributing guide

Research direction

Start with the minimal SQL reproduction and compare both EXPLAIN plans, focusing on the subquery optimization behavior around double negation and IN predicates. Trace the optimizer path that handles these predicates and verify that the equivalent queries receive comparable semi-join and null-rejection treatment. Done means the plans and execution times no longer show the reported regression while both queries return COUNT(*) = 1000.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.