TiDB retains redundant DISTINCT aggregations in set-operation branches
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
TiDB correctly eliminates a redundant `DISTINCT` above a complete `UNION`, but
does not eliminate `DISTINCT` inside the individual inputs of `UNION`,
`INTERSECT`, or `EXCEPT`.
### 1. Minimal reproduce step (Required)
```sql
DROP DATABASE IF EXISTS tidb_set_branch_distinct;
CREATE DATABASE tidb_set_branch_distinct;
USE tidb_set_branch_distinct;
CREATE TABLE digits(d INT PRIMARY KEY);
INSERT INTO digits VALUES(0),(1),(2),(3),(4),(5),(6),(7),(8),(9);
CREATE TABLE lhs(id INT, v INT);
CREATE TABLE rhs(id INT, v INT);
INSERT INTO lhs SELECT n,MOD(n,1000) FROM
(SELECT a.d+b.d*10+c.d*100+d.d*1000+e.d*10000+1 n
FROM digits a,digits b,digits c,digits d,digits e) numbers;
INSERT INTO rhs SELECT n,MOD(n,1000) FROM
(SELECT a.d+b.d*10+c.d*100+d.d*1000+e.d*10000+50001 n
FROM digits a,digits b,digits c,digits d,digits e) numbers;
ANALYZE TABLE lhs,rhs;
EXPLAIN SELECT COUNT(*) FROM
((SELECT DISTINCT id FROM lhs) UNION (SELECT DISTINCT id FROM rhs)) s;
EXPLAIN SELECT COUNT(*) FROM
((SELECT DISTINCT id FROM lhs) INTERSECT (SELECT DISTINCT id FROM rhs)) s;
EXPLAIN SELECT COUNT(*) FROM
((SELECT DISTINCT id FROM lhs) EXCEPT (SELECT DISTINCT id FROM rhs)) s;
SELECT COUNT(*) FROM ((SELECT id FROM lhs) UNION (SELECT id FROM rhs)) s;
SELECT COUNT(*) FROM
((SELECT DISTINCT id FROM lhs) UNION (SELECT DISTINCT id FROM rhs)) s;
SELECT COUNT(*) FROM ((SELECT id FROM lhs) INTERSECT (SELECT id FROM rhs)) s;
SELECT COUNT(*) FROM
((SELECT DISTINCT id FROM lhs) INTERSECT (SELECT DISTINCT id FROM rhs)) s;
SELECT COUNT(*) FROM ((SELECT id FROM lhs) EXCEPT (SELECT id FROM rhs)) s;
SELECT COUNT(*) FROM
((SELECT DISTINCT id FROM lhs) EXCEPT (SELECT DISTINCT id FROM rhs)) s;
```
### 2. What did you expect to see? (Required)
Branch-level `DISTINCT` should be absorbed into the set operation.
### 3. What did you see instead (Required)
For `UNION`, the plan contains a set-level `HashAgg` and an additional
`HashAgg` for each input. `INTERSECT` and `EXCEPT` similarly retain input-side
aggregations before the semi/anti-semi join.
| Operation | Without branch DISTINCT | With branch DISTINCT | Slowdown |
|---|---:|---:|---:|
| UNION | 38.89 ms | 55.36 ms | 1.42x |
| INTERSECT | 25.72 ms | 34.14 ms | 1.33x |
| EXCEPT | 24.61 ms | 34.16 ms | 1.39x |
### 4. What is your TiDB version? (Required)
```text
Release Version: v8.5.7
Edition: Community
Git Commit Hash: 202b7f47286a1109b5c957401d34c9358d130ae0
Git Branch: HEAD
UTC Build Time: 2026-07-15 02:06:00
GoVersion: go1.25.10
Race Enabled: false
Check Table Before Drop: false
Store: unistore
```
Contributor guide
Research direction
Start by running the supplied SQL reproducer and comparing the EXPLAIN plans for UNION, INTERSECT, and EXCEPT with and without branch-level DISTINCT. Trace the set-operation planning path in the Go code, then verify that branch DISTINCT aggregations are absent while results and the reported performance behavior remain correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100