pingcap / pingcap/tidb

TiDB returns incorrect JOIN results when explicit COLLATE is applied to a string expression

Open
#70,860 5 comments 0 reactions 0 assignees View on GitHub
affects-25.10 affects-26.3 affects-7.5 affects-8.1 affects-8.5 contribution first-time-contributor severity/critical sig/execution type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce step

Environment: TiDB v8.5.8, single-node TiUP playground using TiKV storage. The tables below contain no indexes and one row each.

Run the following SQL in a fresh TiDB test database:

```sql
SELECT tidb_version();

CREATE DATABASE IF NOT EXISTS tidb_collate_join_repro;
USE tidb_collate_join_repro;

DROP TABLE IF EXISTS x, y;

CREATE TABLE x (
c VARCHAR(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin
);

CREATE TABLE y (
c VARCHAR(10) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci
);

INSERT INTO x VALUES ('a');
INSERT INTO y VALUES ('A');

SELECT COUNT(*) AS result
FROM x JOIN y
ON CONVERT(x.c USING utf8mb4) COLLATE utf8mb4_unicode_ci = y.c;
```

The version output from the tested TiDB instance was:

```text
Release Version: v8.5.8
Edition: Community
Git Branch: HEAD
GoVersion: go1.25.12
Race Enabled: false
Check Table Before Drop: false
Store: tikv
```

### 2. What did you expect to see?

The query should return `1`.

The explicit `COLLATE utf8mb4_unicode_ci` is case-insensitive, so the values `'a'` and `'A'` should compare equal. MySQL 8.0 and the other tested MySQL-compatible databases also return `1` for this query.

### 3. What did you see instead

TiDB v8.5.8 returns `0`, which incorrectly omits the matching row from the JOIN.

The same TiDB instance returns `1` for the following controls:

```sql
SELECT COUNT(*) AS result
FROM x JOIN y
ON x.c COLLATE utf8mb4_unicode_ci = y.c;
```

```sql
SELECT COUNT(*) AS result
FROM x JOIN y
ON CONVERT(x.c USING utf8mb4) COLLATE utf8mb4_unicode_ci
= y.c COLLATE utf8mb4_unicode_ci;
```

```sql
SELECT COUNT(*) AS result
FROM x
WHERE CONVERT(c USING utf8mb4) COLLATE utf8mb4_unicode_ci = 'A';
```

All three controls return `1` on TiDB v8.5.8.

The failure is specific to using a string expression with an explicit collation as a JOIN comparison operand. Follow-up checks found the same pattern with several expressions:

| JOIN expression on the left | MySQL 8.4.7 | TiDB v8.5.8 |
| --- | ---: | ---: |
| `x.c COLLATE utf8mb4_unicode_ci` | 1 | 1 |
| `CONVERT(x.c USING utf8mb4) COLLATE utf8mb4_unicode_ci` | 1 | 0 |
| `CONCAT(x.c, '') COLLATE utf8mb4_unicode_ci` | 1 | 0 |
| `SUBSTRING(x.c, 1) COLLATE utf8mb4_unicode_ci` | 1 | 0 |
| `IFNULL(x.c, '') COLLATE utf8mb4_unicode_ci` | 1 | 0 |
| `LOWER(x.c) COLLATE utf8mb4_unicode_ci` | 1 | 0 |

The issue is reproducible without indexes or a large data set. `EXPLAIN` uses a HashJoin in the minimal case. Forcing another join algorithm with TiDB optimizer hints did not change the incorrect result in a follow-up check.

Cross-database comparison for the minimal query:

| Database | Result |
| --- | ---: |
| MySQL 8.0.46 | 1 |
| MySQL 8.4.7 | 1 |
| MariaDB 12.3.2 | 1 |
| Percona Server 8.4.11-11 | 1 |
| GreatSQL 8.4.4-5 | 1 |
| TiDB v8.5.8 | 0 |

This appears to be a collation propagation or comparison-key issue in JOIN execution for non-column string expressions. I have not confirmed the exact internal root cause.

### 4. What is your TiDB version?

```text
Release Version: v8.5.8
Edition: Community
Git Branch: HEAD
GoVersion: go1.25.12
Race Enabled: false
Check Table Before Drop: false
Store: tikv
```

Contributor guide

Open the contributing guide

Research direction

Start by running the minimal SQL reproduction and its control queries, then inspect the HashJoin path and how collation is propagated for non-column string expressions. The work is done when the JOIN returns 1 for the explicit COLLATE cases, without regressing the controls or changing the result across join algorithms.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.