Inconsistent `GROUP BY` on `NULL`s Due to Expression Rewriting When Pushed Down to TiKV
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
Please answer these questions before submitting your issue. Thanks!
### 1. Minimal Reproduce Step (Required)
```sql
DROP DATABASE IF EXISTS repro_tidb617_db31_min;
CREATE DATABASE repro_tidb617_db31_min;
USE repro_tidb617_db31_min;
SET SESSION sql_mode='ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';
SET time_zone='+00:00';
CREATE TABLE src(
id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY,
c0 BOOLEAN NULL,
c2 DECIMAL(65,30) NULL
);
INSERT INTO src(c0,c2) VALUES
(0,0),
(1,NULL);
CREATE TABLE l(
id BIGINT NOT NULL PRIMARY KEY,
c2 DECIMAL(65,30) NULL
);
CREATE TABLE r(
id BIGINT NOT NULL PRIMARY KEY,
c0 BOOLEAN NULL
);
INSERT INTO l SELECT id,c2 FROM src;
INSERT INTO r SELECT id,c0 FROM src;
-- Single‑table query (returns 2 rows – wrong)
SELECT
CAST(
CASE '2025-02-01 20:39:39'
WHEN c0 THEN ((false) REGEXP NULL)
ELSE CAST(c2 AS TIME)
END AS DATETIME
) AS val
FROM src
GROUP BY
CAST(
CASE '2025-02-01 20:39:39'
WHEN c0 THEN ((false) REGEXP NULL)
ELSE CAST(c2 AS TIME)
END AS DATETIME
);
-- Split/join query (returns 1 row – correct)
SELECT
CAST(
CASE '2025-02-01 20:39:39'
WHEN v.c0 THEN ((false) REGEXP NULL)
ELSE CAST(v.c2 AS TIME)
END AS DATETIME
) AS val
FROM (
SELECT l.id, r.c0, l.c2
FROM l
JOIN (
SELECT id, COUNT(*) AS cnt
FROM r
GROUP BY id
) rg ON l.id = rg.id
JOIN r ON r.id = rg.id
) v
GROUP BY
CAST(
CASE '2025-02-01 20:39:39'
WHEN v.c0 THEN ((false) REGEXP NULL)
ELSE CAST(v.c2 AS TIME)
END AS DATETIME
);
```
### 2. What Did You Expect to See? (Required)
For both rows in `src`, the `CASE` expression always returns `CAST(c2 AS TIME)`.
- Row 1 (`c2=0`): `CAST(0 AS TIME)` → `'00:00:00'`, then `CAST('00:00:00' AS DATETIME)` returns `NULL` (invalid conversion).
- Row 2 (`c2=NULL`): `CAST(NULL AS TIME)` → `NULL`, then `CAST(NULL AS DATETIME)` → `NULL`.
Therefore both rows produce `NULL` as the grouping key. `GROUP BY` on `NULL` should merge them into a single group, and the query should return `1 row` with `val = NULL`.
Both the single‑table and the join query should return the same result: `1 row`.
### 3. What Did You See Instead? (Required)
- Single‑table query: returns `2 rows` (both `NULL`) — wrong.
- Split/join query: returns `1 row` (`NULL`) — correct.
### 4. What Is Your TiDB Version? (Required)
Version: TiDB‑v9.0.0
### 5. Execution Plan Differences
**Single‑table (wrong)** — `GROUP BY` pushed down to TiKV coprocessor with an extra type cast:
```
HashAgg_5 [cop[tikv]]
group by:cast(
case(
eq(2025, cast(src.c0, double)),
NULL,
cast(cast(src.c2, time), var_string(10)) ← extra var_string cast
),
datetime
)
```
The `ELSE` branch is rewritten from `CAST(c2 AS TIME)` to `CAST(CAST(c2 AS TIME), VAR_STRING(10))`. This extra cast changes the internal representation of the resulting `NULL`, making the two rows produce different group keys at TiKV.
**Split/join (correct)** — `GROUP BY` remains at TiDB root, expression unchanged:
```
HashAgg_20 [root]
group by:Column#16 ...
└─Projection_87 [root]
cast(case(...), datetime)->Column#16 ← original expression, no extra cast
```
### 6. Root Cause
When the optimizer pushes a `GROUP BY` containing a `CASE ... CAST(... AS TIME) ... END AS DATETIME` expression down to TiKV, it unnecessarily wraps the inner `CAST(c2 AS TIME)` with an additional `CAST(... AS VAR_STRING(10))`. This alters the datatype of the intermediate value:
- Without the extra cast, `CAST(0 AS TIME)` → `'00:00:00'` → outer `CAST AS DATETIME` → `NULL`.
- With the extra `VAR_STRING(10)` cast applied inside TiKV, the `NULL` value that arises from the outer `CAST AS DATETIME` gets a different internal representation compared to the `NULL` from the `WHEN c0` branch (which remains unchanged). As a result, the two rows are no longer considered equal by TiKV's `GROUP BY` logic, and they remain separate groups.
Contributor guide
Research direction
Reproduce the supplied SQL in TiDB v9.0.0 and compare the single-table and split/join execution plans. Trace the optimizer's GROUP BY pushdown and expression rewriting for the CASE expression involving TIME and DATETIME casts; done means both queries return one NULL row without producing divergent grouping keys in TiKV.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100