pingcap / pingcap/tidb

With pseudo statistics, GROUP BY / DISTINCT over COALESCE(float_col) collapses rows into wrong groups and mis-renders the keys (two-level aggregate re-encodes the pushed-down key)

Open
#70,754 4 comments 0 reactions 0 assignees View on GitHub
affects-25.10 affects-26.3 affects-7.5 affects-8.1 affects-8.5 contribution severity/critical sig/execution type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

```sql
DROP DATABASE IF EXISTS repro_coalesce_float;
CREATE DATABASE repro_coalesce_float;
USE repro_coalesce_float;
CREATE TABLE t0(c0 FLOAT);
INSERT INTO t0 VALUES (0.06034067),(0.1236711),(0.14868417),(0.15180263),(0.24330534),
(0.28414282),(0.70095146),(0.7055481),(0.7891293),(0.7985108),(0.8217786),(0.97570187),
(1249306200);
CREATE OR REPLACE VIEW v(c0) AS SELECT COALESCE(t0.c0) FROM t0;
-- do NOT run ANALYZE: with pseudo statistics the planner picks the two-level
-- (cop + root) aggregate; with up-to-date stats it picks a single-level aggregate
-- and the bug does not reproduce (details in Section 3)

-- Base query (predicate over the GROUP BY key in the WHERE clause):
SELECT v.c0 FROM v WHERE (v.c0) GROUP BY v.c0;
-- -3.68935e19
-- -0
-- -2
-- 2
-- -1.0842e-19
-- 3.68935e19
-- 1.0842e-19
-- 7 rows <-- WRONG twice over: (1) the '-0' group's boolean value is FALSE, so the
-- WHERE filter must drop it — it survives here; (2) ALL group keys are
-- garbage reinterpretations of the FLOAT bit patterns, not the stored values

-- Rewritten query (same key materialized into the derived-table projection):
SELECT ref0 FROM (SELECT v.c0 AS ref0, (v.c0) AS ref1 FROM v GROUP BY v.c0) s WHERE ref1;
-- 6 rows <-- the '-0' group is correctly filtered out on this path — the two
-- equivalent forms disagree 7 vs 6 (keys are equally garbage here)
```

Semantics: `COALESCE(c0)` is an identity function on a non-NULL column, so
`GROUP BY v.c0` must produce exactly the stored distinct values as group keys — instead
the keys come out as garbage reinterpretations of the FLOAT bit patterns
(`±1.08e-19`, `±3.69e19`, `±2`, `-0`), and the two equivalent query forms disagree
(7 vs 6) because the garbage `-0` key's boolean coercion differs between the
WHERE-position and the group-representative position.

The same defect in its smallest form (2 rows — every fresh table reproduces, verified
twice end-to-end):

```sql
DROP DATABASE IF EXISTS repro_coalesce_min;
CREATE DATABASE repro_coalesce_min;
USE repro_coalesce_min;
CREATE TABLE t(c0 FLOAT);
INSERT INTO t VALUES (1),(2);
-- no ANALYZE (same reason as above)

-- Base query:
SELECT COALESCE(t.c0) FROM t WHERE (COALESCE(t.c0)) GROUP BY COALESCE(t.c0);
-- 1 <-- WRONG (expected: 2 rows, 1 and 2)

-- Rewritten query:
SELECT ref0 FROM (SELECT COALESCE(t.c0) AS ref0, COALESCE(t.c0) AS ref1 FROM t GROUP BY COALESCE(t.c0)) s WHERE ref1;
-- 1 <-- WRONG too (both rows collapse into one group)

-- The collapse is provable without any pair: MIN != MAX inside ONE output row
SELECT MIN(c0), MAX(c0), COUNT(*) FROM t GROUP BY COALESCE(c0);
-- 1 | 2 | 2 <-- two different values judged equal as group keys
SELECT DISTINCT COALESCE(c0) FROM t;
-- 0 <-- a value that exists in neither row
```

Verified trigger conditions (single-variable experiments):

- **Plan-state dependency** (the load-bearing condition): pseudo statistics →
two-level aggregate → deterministic reproduction on every freshly created table;
`ANALYZE TABLE` → single-level root HashAgg → correct results (EXPLAIN in Section 3).
- Only FLOAT (signed and unsigned) is affected. The identical queries on `DOUBLE`,
`DECIMAL(10,2)`, `INT UNSIGNED`, and `VARCHAR(10)` all return the correct N groups.
- `GROUP BY c0` (bare column key) is correct — the expression wrapper on the grouping
key is load-bearing.
- NoREC does not reproduce (measured live on both repro databases; see Section 3).

Found by SQLancer's DQR oracle (query-relocation differential testing) on TiDB nightly:
a 44-statement generated test case produced the 7-vs-6 disagreement above; minimized by
hand to the 13-row view form and the 2-row absolute form shown here.

Not a duplicate: no previously reported issue in our tracked set covers `COALESCE` on a
FLOAT grouping key under two-level aggregation (DOUBLE/DECIMAL/INT/VARCHAR are unaffected,
so this is not the generic "expression in GROUP BY" family either).
### 2. What did you expect to see? (Required)
`COALESCE(c0)` is an identity function on a non-NULL column, so `GROUP BY COALESCE(c0)`
must produce exactly the same groups as `GROUP BY c0` (the distinct stored values),
and `SELECT DISTINCT COALESCE(c0)` must return the stored values — on every plan, with
or without cop-level pre-aggregation.

### 3. What did you see instead (Required)
With pseudo statistics the planner builds a two-level aggregate; the grouping keys
produced by the cop-level pre-aggregation are mis-judged when the root-level aggregate
re-groups the pre-aggregated partial results: distinct values collapse into shared
groups, the rendered keys come out as garbage reinterpretations of the FLOAT bit
patterns (`±1.08e-19`, `±3.69e19`, `±2`, `-0`), `DISTINCT` returns a value that exists
in no row, and equivalent query forms disagree (7 vs 6) over whether the garbage `-0`
group survives. Rows are silently dropped from aggregation results.

`EXPLAIN SELECT COALESCE(c0) AS v, COUNT(*) AS cnt FROM t GROUP BY COALESCE(c0);`
(the wrong plan — note the cop-level `HashAgg_5` under the root-level `HashAgg_9`)

```
id estRows task access object operator info
Projection_4 8000.00 root coalesce(repro_coalesce_float.t.c0)->Column#5, Column#4
└─HashAgg_9 8000.00 root group by:Column#7, funcs:count(Column#8)->Column#4, funcs:firstrow(Column#9)->repro_coalesce_float.t.c0
└─TableReader_10 8000.00 root data:HashAgg_5
└─HashAgg_5 8000.00 cop[tikv] group by:coalesce(repro_coalesce_float.t.c0), funcs:count(1)->Column#8, funcs:firstrow(repro_coalesce_float.t.c0)->Column#9
└─TableFullScan_8 10000.00 cop[tikv] table:t, keep order:false, stats:pseudo
```

After `ANALYZE TABLE t` the same query plans as a single root-level HashAgg fed by a
root Projection (no cop-level aggregate) and returns the correct groups — which isolates
the defect to the cop→root handoff of the pushed-down `coalesce(FLOAT)` grouping key:
the partial pre-aggregation results keyed on that expression are re-grouped incorrectly
at the root level.
### 4. What is your TiDB version? (Required)

```
Release Version: v9.0.0-beta.2.pre-2174-g65ac2fad58
Edition: Community
Git Commit Hash: 65ac2fad582510b92d3c4b79999e48217c989737
Git Branch: HEAD
UTC Build Time: 2026-08-28 20:22:53
GoVersion: go1.25.12
Race Enabled: false
Check Table Before Drop: false
Store: unistore
Kernel Type: Classic
```

Contributor guide

Open the contributing guide

Research direction

Run the two-row SQL reproduction and EXPLAIN without ANALYZE first; focus on the cop[tikv] HashAgg_5 to root HashAgg_9 handoff for the pushed-down COALESCE(FLOAT) grouping key. Done when GROUP BY and DISTINCT preserve the stored distinct values and the base and rewritten forms agree under pseudo statistics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.