GBK in TiDB is incompatible with that in MySQL
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce step (Required)
```sql
SELECT CONVERT(CAST(0xA2E3 AS CHAR CHARACTER SET gbk) USING utf8mb4);
SELECT CONVERT(CAST(0xA2E4 AS CHAR CHARACTER SET gbk) USING utf8mb4);
```
### 2. What did you expect to see? (Required)
(If it is fully compatible with MySQL)
Both should be:
```
+---------------------------------------------------------------+
| CONVERT(CAST(0xA2E3 AS CHAR CHARACTER SET gbk) USING utf8mb4) |
+---------------------------------------------------------------+
| ? |
+---------------------------------------------------------------+
```
### 3. What did you see instead (Required)
The first one:
```
+---------------------------------------------------------------+
| CONVERT(CAST(0xA2E3 AS CHAR CHARACTER SET gbk) USING utf8mb4) |
+---------------------------------------------------------------+
| € |
+---------------------------------------------------------------+
1 row in set (0.001 sec)
```
The second one (with warning):
```
+---------------------------------------------------------------+
| CONVERT(CAST(0xA2E4 AS CHAR CHARACTER SET gbk) USING utf8mb4) |
+---------------------------------------------------------------+
| NULL |
+---------------------------------------------------------------+
1 row in set, 1 warning (0.001 sec)
MySQL [demo_tidb]> SHOW WARNINGS;
+---------+------+-----------------------------------------------------+
| Level | Code | Message |
+---------+------+-----------------------------------------------------+
| Warning | 3854 | Cannot convert string '\xA2\xE4' from binary to gbk |
+---------+------+-----------------------------------------------------+
1 row in set (0.001 sec)
```
### 4. What is your TiDB version? (Required)
```
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tidb_version() |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Release Version: v8.5.0
Edition: Community
Git Commit Hash: d13e52ed6e22cc5789bed7c64c861578cd2ed55b
Git Branch: HEAD
UTC Build Time: 2024-12-18 02:26:06
GoVersion: go1.23.3
Race Enabled: false
Check Table Before Drop: false
Store: unistore |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
## More Info
In TiDB,
```
> SELECT @@sql_mode, @@character_set_client, @@character_set_connection, @@character_set_results;
+-------------------------------------------------------------------------------------------------------------------------------------------+------------------------+----------------------------+-------------------------+
| @@sql_mode | @@character_set_client | @@character_set_connection | @@character_set_results |
+-------------------------------------------------------------------------------------------------------------------------------------------+------------------------+----------------------------+-------------------------+
| ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION | utf8mb4 | utf8mb4 | utf8mb4 |
+-------------------------------------------------------------------------------------------------------------------------------------------+------------------------+----------------------------+-------------------------+
```
In MySQL,
```
+-----------------------------------------------------------------------------------------------------------------------+------------------------+----------------------------+-------------------------+
| @@sql_mode | @@character_set_client | @@character_set_connection | @@character_set_results |
+-----------------------------------------------------------------------------------------------------------------------+------------------------+----------------------------+-------------------------+
| ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION | utf8mb4 | utf8mb4 | utf8mb4 |
+-----------------------------------------------------------------------------------------------------------------------+------------------------+----------------------------+-------------------------+
```
In fact, there many such divergences,
Summary:
- mismatched mappings: 82
- present only in mysql_valid_gbk_utf8_list.txt: 2067
- present only in tidb_valid_gbk.txt: 0
I know that there is no *single* standard in terms of character set mappings, but TiDB declares that it is compatible with MySQL. The divergences are not mentioned in [character-set-gbk](https://docs.pingcap.com/tidb/stable/character-set-gbk/). So it can be seen as bugs.
Contributor guide
Research direction
Start by reproducing the two GBK conversion queries and compare their results with MySQL. Then inspect the GBK mapping data, including mysql_valid_gbk_utf8_list.txt and tidb_valid_gbk.txt, to identify the compatibility gaps; done means the reported mappings match MySQL and the divergences are covered by regression tests or documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, mysql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100