INSERT skips VARCHAR/CHAR length truncation when charset validation fires (utf8mb4 into utf8, non-strict mode)
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report
### 1. Minimal reproduce step (Required)
```sql
-- Tested on v8.5.1 and v8.5.5
-- 1. Non-strict sql_mode (no STRICT_TRANS_TABLES)
SET SESSION sql_mode = 'ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,ALLOW_INVALID_DATES';
-- 2. Table with utf8 charset column
CREATE TABLE test_varchar_bypass (
id INT AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(10) COLLATE utf8_general_ci DEFAULT NULL
);
-- 3. Insert a string with utf8mb4 characters (emoji) that exceeds VARCHAR(10)
-- This is 30 characters: 26 ASCII + 4 emoji
INSERT INTO test_varchar_bypass (name)
VALUES ('ABCDE📝FGHIJ📓KLMNO🖍PQRST📝UVWXYZ');
-- 4. Check what was stored
SELECT id, CHAR_LENGTH(name) AS char_len, name FROM test_varchar_bypass;
```
**What triggers it:** Inserting utf8mb4 characters (e.g. emoji) into a `utf8` column when `STRICT_TRANS_TABLES` is absent from `sql_mode`.
TiDB replaces each invalid 4-byte sequence with `?` (Warning 1366), but **skips the VARCHAR/CHAR length truncation step**. The full replacement string is stored regardless of column limit.
### 2. What did you expect to see? (Required)
MySQL 8.0 behavior — replace invalid bytes AND truncate to VARCHAR(10):
```
+----+----------+------------+
| id | char_len | name |
+----+----------+------------+
| 1 | 10 | ABCDE?FGHI |
+----+----------+------------+
-- Warning 1366: Incorrect string value (charset)
-- Warning 1265: Data truncated for column 'name' (length)
```
### 3. What did you see instead (Required)
TiDB stores the full 30-character string in a VARCHAR(10) column:
```
+----+----------+--------------------------------+
| id | char_len | name |
+----+----------+--------------------------------+
| 1 | 30 | ABCDE?FGHIJ?KLMNO?PQRST?UVWXYZ |
+----+----------+--------------------------------+
-- Warning 1366 fires, but Warning 1265 does NOT (no truncation)
```
### 4. What is your TiDB version? (Required)
Reproduced on v8.5.1 and v8.5.5. Likely affects earlier versions — the code path has been stable across releases.
```
mysql> SELECT tidb_version()\G
Release Version: v8.5.1
Edition: Community
Git Commit Hash: ...
```
### Scope and impact
**Affected paths:** All INSERT-family operations — INSERT, REPLACE INTO, INSERT...SELECT, LOAD DATA, PREPARE/EXECUTE. Also affects CHAR(N) columns, not just VARCHAR. UPDATE and ON DUPLICATE KEY UPDATE are **not affected** — they handle charset conversion before length enforcement, so both steps run.
**Three conditions required (all must be true):**
1. Column charset is `utf8` (not `utf8mb4`)
2. `STRICT_TRANS_TABLES` absent from `sql_mode`
3. Input contains utf8mb4 data (emoji, CJK Extension B, musical symbols, etc.)
**No limit on stored length:** `REPEAT('A📝', 5000)` stores 10,000 characters in VARCHAR(10) (test E25). The column length constraint is entirely unenforced.
**Indexes:** Secondary indexes store the oversized data. Unique indexes accept two different oversized values that share the same VARCHAR(10)-length prefix. `ADMIN CHECK TABLE` does **not** detect the violation — no built-in detection exists after the fact.
**Production impact:** VARCHAR(70) column storing up to 141 characters in production. Does not break reads or writes, but causes `ERROR 1406: Data Too Long` during cluster rebuild via Dumpling + IMPORT INTO (which defaults to strict mode). Usually discovered only during migration or disaster recovery.
**MySQL 8.0 comparison:** 15 side-by-side tests — MySQL correctly truncates in all 14 scenarios where TiDB bypasses, and rejects with ERROR 3988 in the remaining one.
**Workarounds:**
- `SET GLOBAL tidb_check_mb4_value_in_utf8 = OFF` (narrowest)
- `SET GLOBAL tidb_skip_utf8_check = ON` (broader)
- Use `utf8mb4` column charset instead of `utf8`
- Use `STRICT_TRANS_TABLES` in `sql_mode`
Full lab with 196 tests across 9 phases, including MySQL 8.0 comparison: [lab-07-varchar-length-enforcement](https://github.com/alastori/tidb-sandbox/blob/main/labs/tidb/lab-07-varchar-length-enforcement/lab-07-varchar-length-enforcement.md)
### Root cause hint
In `pkg/types/datum.go`, `convertToString` (~line 1207):
```go
if err == nil {
s, err = ProduceStrWithSpecifiedTp(s, target, ctx, true)
}
```
`GetStringWithCheck` returns a charset error (Warning 1366). The `if err == nil` gate prevents `ProduceStrWithSpecifiedTp` (the truncation function) from running. The replacement string is stored as-is, bypassing length enforcement.
### Follow-up questions
- `ADMIN CHECK TABLE` currently does not check `CHAR_LENGTH(col)` against declared column limits — may warrant a separate enhancement
- Backup/restore and replication paths (Dumpling, IMPORT INTO, TiCDC, BR) may need evaluation for clusters that already have oversized rows
### Related issues
- #64711 — Recursive CTE truncation bypass (same class: truncation skipped on a specific code path)
- #60330 — Memtables skip type checking
- #65323 / #63775 — Inconsistent Data Too Long enforcement on other paths
Contributor guide
Assessment
This issue has not been assessed yet.