pingcap / pingcap/tidb

GBK in TiDB is incompatible with that in MySQL

Open
#66,126 6 comments 1 reaction 0 assignees View on GitHub
contribution severity/moderate sig/sql-infra type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

### 1. Minimal reproduce step (Required)

```sql
SELECT CONVERT(CAST(0xA2E3 AS CHAR CHARACTER SET gbk) USING utf8mb4);

SELECT CONVERT(CAST(0xA2E4 AS CHAR CHARACTER SET gbk) USING utf8mb4);
```

### 2. What did you expect to see? (Required)

(If it is fully compatible with MySQL)

Both should be:

```
+---------------------------------------------------------------+
| CONVERT(CAST(0xA2E3 AS CHAR CHARACTER SET gbk) USING utf8mb4) |
+---------------------------------------------------------------+
| ? |
+---------------------------------------------------------------+
```

### 3. What did you see instead (Required)

The first one:

```
+---------------------------------------------------------------+
| CONVERT(CAST(0xA2E3 AS CHAR CHARACTER SET gbk) USING utf8mb4) |
+---------------------------------------------------------------+
| € |
+---------------------------------------------------------------+
1 row in set (0.001 sec)
```

The second one (with warning):

```
+---------------------------------------------------------------+
| CONVERT(CAST(0xA2E4 AS CHAR CHARACTER SET gbk) USING utf8mb4) |
+---------------------------------------------------------------+
| NULL |
+---------------------------------------------------------------+
1 row in set, 1 warning (0.001 sec)

MySQL [demo_tidb]> SHOW WARNINGS;
+---------+------+-----------------------------------------------------+
| Level | Code | Message |
+---------+------+-----------------------------------------------------+
| Warning | 3854 | Cannot convert string '\xA2\xE4' from binary to gbk |
+---------+------+-----------------------------------------------------+
1 row in set (0.001 sec)
```

### 4. What is your TiDB version? (Required)

```
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version() |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v8.5.0
Edition: Community
Git Commit Hash: d13e52ed6e22cc5789bed7c64c861578cd2ed55b
Git Branch: HEAD
UTC Build Time: 2024-12-18 02:26:06
GoVersion: go1.23.3
Race Enabled: false
Check Table Before Drop: false
Store: unistore |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```

## More Info

In TiDB,

```
> SELECT @@sql_mode, @@character_set_client, @@character_set_connection, @@character_set_results;
+-------------------------------------------------------------------------------------------------------------------------------------------+------------------------+----------------------------+-------------------------+
| @@sql_mode | @@character_set_client | @@character_set_connection | @@character_set_results |
+-------------------------------------------------------------------------------------------------------------------------------------------+------------------------+----------------------------+-------------------------+
| ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION | utf8mb4 | utf8mb4 | utf8mb4 |
+-------------------------------------------------------------------------------------------------------------------------------------------+------------------------+----------------------------+-------------------------+
```

In MySQL,

```
+-----------------------------------------------------------------------------------------------------------------------+------------------------+----------------------------+-------------------------+
| @@sql_mode | @@character_set_client | @@character_set_connection | @@character_set_results |
+-----------------------------------------------------------------------------------------------------------------------+------------------------+----------------------------+-------------------------+
| ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION | utf8mb4 | utf8mb4 | utf8mb4 |
+-----------------------------------------------------------------------------------------------------------------------+------------------------+----------------------------+-------------------------+
```

In fact, there many such divergences,

Summary:
- mismatched mappings: 82
- present only in mysql_valid_gbk_utf8_list.txt: 2067
- present only in tidb_valid_gbk.txt: 0

I know that there is no *single* standard in terms of character set mappings, but TiDB declares that it is compatible with MySQL. The divergences are not mentioned in [character-set-gbk](https://docs.pingcap.com/tidb/stable/character-set-gbk/). So it can be seen as bugs.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two GBK conversion queries and compare their results with MySQL. Then inspect the GBK mapping data, including mysql_valid_gbk_utf8_list.txt and tidb_valid_gbk.txt, to identify the compatibility gaps; done means the reported mappings match MySQL and the divergences are covered by regression tests or documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, mysql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.