pingcap / pingcap/tidb

Inconsistent `GROUP BY` on `NULL`s Due to Expression Rewriting When Pushed Down to TiKV

Open
#69,358 2 comments 0 reactions 0 assignees View on GitHub
contribution may-affects-7.5 may-affects-8.1 may-affects-8.5 severity/critical sig/planner type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal Reproduce Step (Required)

```sql
DROP DATABASE IF EXISTS repro_tidb617_db31_min;
CREATE DATABASE repro_tidb617_db31_min;
USE repro_tidb617_db31_min;

SET SESSION sql_mode='ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';
SET time_zone='+00:00';

CREATE TABLE src(
id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY,
c0 BOOLEAN NULL,
c2 DECIMAL(65,30) NULL
);

INSERT INTO src(c0,c2) VALUES
(0,0),
(1,NULL);

CREATE TABLE l(
id BIGINT NOT NULL PRIMARY KEY,
c2 DECIMAL(65,30) NULL
);

CREATE TABLE r(
id BIGINT NOT NULL PRIMARY KEY,
c0 BOOLEAN NULL
);

INSERT INTO l SELECT id,c2 FROM src;
INSERT INTO r SELECT id,c0 FROM src;

-- Single‑table query (returns 2 rows – wrong)
SELECT
CAST(
CASE '2025-02-01 20:39:39'
WHEN c0 THEN ((false) REGEXP NULL)
ELSE CAST(c2 AS TIME)
END AS DATETIME
) AS val
FROM src
GROUP BY
CAST(
CASE '2025-02-01 20:39:39'
WHEN c0 THEN ((false) REGEXP NULL)
ELSE CAST(c2 AS TIME)
END AS DATETIME
);

-- Split/join query (returns 1 row – correct)
SELECT
CAST(
CASE '2025-02-01 20:39:39'
WHEN v.c0 THEN ((false) REGEXP NULL)
ELSE CAST(v.c2 AS TIME)
END AS DATETIME
) AS val
FROM (
SELECT l.id, r.c0, l.c2
FROM l
JOIN (
SELECT id, COUNT(*) AS cnt
FROM r
GROUP BY id
) rg ON l.id = rg.id
JOIN r ON r.id = rg.id
) v
GROUP BY
CAST(
CASE '2025-02-01 20:39:39'
WHEN v.c0 THEN ((false) REGEXP NULL)
ELSE CAST(v.c2 AS TIME)
END AS DATETIME
);
```

### 2. What Did You Expect to See? (Required)

For both rows in `src`, the `CASE` expression always returns `CAST(c2 AS TIME)`.

- Row 1 (`c2=0`): `CAST(0 AS TIME)` → `'00:00:00'`, then `CAST('00:00:00' AS DATETIME)` returns `NULL` (invalid conversion).
- Row 2 (`c2=NULL`): `CAST(NULL AS TIME)` → `NULL`, then `CAST(NULL AS DATETIME)` → `NULL`.

Therefore both rows produce `NULL` as the grouping key. `GROUP BY` on `NULL` should merge them into a single group, and the query should return `1 row` with `val = NULL`.

Both the single‑table and the join query should return the same result: `1 row`.

### 3. What Did You See Instead? (Required)

- Single‑table query: returns `2 rows` (both `NULL`) — wrong.
- Split/join query: returns `1 row` (`NULL`) — correct.

### 4. What Is Your TiDB Version? (Required)

Version: TiDB‑v9.0.0

### 5. Execution Plan Differences

**Single‑table (wrong)** — `GROUP BY` pushed down to TiKV coprocessor with an extra type cast:

```
HashAgg_5 [cop[tikv]]
group by:cast(
case(
eq(2025, cast(src.c0, double)),
NULL,
cast(cast(src.c2, time), var_string(10)) ← extra var_string cast
),
datetime
)
```

The `ELSE` branch is rewritten from `CAST(c2 AS TIME)` to `CAST(CAST(c2 AS TIME), VAR_STRING(10))`. This extra cast changes the internal representation of the resulting `NULL`, making the two rows produce different group keys at TiKV.

**Split/join (correct)** — `GROUP BY` remains at TiDB root, expression unchanged:

```
HashAgg_20 [root]
group by:Column#16 ...
└─Projection_87 [root]
cast(case(...), datetime)->Column#16 ← original expression, no extra cast
```

### 6. Root Cause

When the optimizer pushes a `GROUP BY` containing a `CASE ... CAST(... AS TIME) ... END AS DATETIME` expression down to TiKV, it unnecessarily wraps the inner `CAST(c2 AS TIME)` with an additional `CAST(... AS VAR_STRING(10))`. This alters the datatype of the intermediate value:

- Without the extra cast, `CAST(0 AS TIME)` → `'00:00:00'` → outer `CAST AS DATETIME` → `NULL`.
- With the extra `VAR_STRING(10)` cast applied inside TiKV, the `NULL` value that arises from the outer `CAST AS DATETIME` gets a different internal representation compared to the `NULL` from the `WHEN c0` branch (which remains unchanged). As a result, the two rows are no longer considered equal by TiKV's `GROUP BY` logic, and they remain separate groups.

Contributor guide

Open the contributing guide

Research direction

Reproduce the supplied SQL in TiDB v9.0.0 and compare the single-table and split/join execution plans. Trace the optimizer's GROUP BY pushdown and expression rewriting for the CASE expression involving TIME and DATETIME casts; done means both queries return one NULL row without producing divergent grouping keys in TiKV.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.