pingcap / pingcap/tidb

TiDB retains redundant DISTINCT aggregations in set-operation branches

Open
#69,997 2 comments 0 reactions 0 assignees View on GitHub
contribution severity/moderate sig/planner type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

TiDB correctly eliminates a redundant `DISTINCT` above a complete `UNION`, but
does not eliminate `DISTINCT` inside the individual inputs of `UNION`,
`INTERSECT`, or `EXCEPT`.
### 1. Minimal reproduce step (Required)

```sql
DROP DATABASE IF EXISTS tidb_set_branch_distinct;
CREATE DATABASE tidb_set_branch_distinct;
USE tidb_set_branch_distinct;
CREATE TABLE digits(d INT PRIMARY KEY);
INSERT INTO digits VALUES(0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
CREATE TABLE lhs(id INT, v INT);
CREATE TABLE rhs(id INT, v INT);
INSERT INTO lhs SELECT n,MOD(n,1000) FROM
(SELECT a.d+b.d*10+c.d*100+d.d*1000+e.d*10000+1 n
FROM digits a,digits b,digits c,digits d,digits e) numbers;
INSERT INTO rhs SELECT n,MOD(n,1000) FROM
(SELECT a.d+b.d*10+c.d*100+d.d*1000+e.d*10000+50001 n
FROM digits a,digits b,digits c,digits d,digits e) numbers;
ANALYZE TABLE lhs,rhs;

EXPLAIN SELECT COUNT(*) FROM
((SELECT DISTINCT id FROM lhs) UNION (SELECT DISTINCT id FROM rhs)) s;
EXPLAIN SELECT COUNT(*) FROM
((SELECT DISTINCT id FROM lhs) INTERSECT (SELECT DISTINCT id FROM rhs)) s;
EXPLAIN SELECT COUNT(*) FROM
((SELECT DISTINCT id FROM lhs) EXCEPT (SELECT DISTINCT id FROM rhs)) s;

SELECT COUNT(*) FROM ((SELECT id FROM lhs) UNION (SELECT id FROM rhs)) s;
SELECT COUNT(*) FROM
((SELECT DISTINCT id FROM lhs) UNION (SELECT DISTINCT id FROM rhs)) s;
SELECT COUNT(*) FROM ((SELECT id FROM lhs) INTERSECT (SELECT id FROM rhs)) s;
SELECT COUNT(*) FROM
((SELECT DISTINCT id FROM lhs) INTERSECT (SELECT DISTINCT id FROM rhs)) s;
SELECT COUNT(*) FROM ((SELECT id FROM lhs) EXCEPT (SELECT id FROM rhs)) s;
SELECT COUNT(*) FROM
((SELECT DISTINCT id FROM lhs) EXCEPT (SELECT DISTINCT id FROM rhs)) s;
```

### 2. What did you expect to see? (Required)
Branch-level `DISTINCT` should be absorbed into the set operation.

### 3. What did you see instead (Required)
For `UNION`, the plan contains a set-level `HashAgg` and an additional
`HashAgg` for each input. `INTERSECT` and `EXCEPT` similarly retain input-side
aggregations before the semi/anti-semi join.

| Operation | Without branch DISTINCT | With branch DISTINCT | Slowdown |
|---|---:|---:|---:|
| UNION | 38.89 ms | 55.36 ms | 1.42x |
| INTERSECT | 25.72 ms | 34.14 ms | 1.33x |
| EXCEPT | 24.61 ms | 34.16 ms | 1.39x |

### 4. What is your TiDB version? (Required)

```text
Release Version: v8.5.7
Edition: Community
Git Commit Hash: 202b7f47286a1109b5c957401d34c9358d130ae0
Git Branch: HEAD
UTC Build Time: 2026-07-15 02:06:00
GoVersion: go1.25.10
Race Enabled: false
Check Table Before Drop: false
Store: unistore
```

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied SQL reproducer and comparing the EXPLAIN plans for UNION, INTERSECT, and EXCEPT with and without branch-level DISTINCT. Trace the set-operation planning path in the Go code, then verify that branch DISTINCT aggregations are absent while results and the reported performance behavior remain correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.