pingcap / pingcap/tidb

Support `latin1_swedish_ci`

Open
#67,198 0 comments 1 reaction 0 assignees View on GitHub
component/charset type/feature-request
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Feature Request

**Is your feature request related to a problem? Please describe:**

[MySQL used `latin1_swedish_ci` as the default collation](https://dev.mysql.com/doc/refman/8.4/en/charset-we-sets.html#:~:text=The%20latin1%5Fswedish%5Fci%20collation%20is%20the%20default%20that%20probably%20is%20used%20by%20the%20majority%20of%20MySQL%20customers%2E) before v8.x. As such many historical databases upgraded from MySQL v5.x retained the `latin1_swedish_ci` columns, despite the developers or users not being Swedish🇸🇪 at all.

This imposes difficulty when migrating such databases into TiDB, because TiDB does not support `latin1_swedish_ci`. They have to change the collation in the downstream TiDB (the most compatible one is `utf8mb4_general_ci`), and this would require the user having to manually change the table schema downstream when using Lightning / DM, and also cross-charset comparison is buggy with sync-diff-inspector.

> Internal references: TCOC-1999, GTOC-6896, GTOC-7520, GTOC-8307

**Describe the feature you'd like:**

Actually implement the `latin1_swedish_ci` collation.

Running this in MySQL we can extract the WEIGHT_STRING mapping:

```sql
with recursive code_points as (
select 0 as c
union all
select c + 1
from code_points
where c < 255
), characters as (
select convert(char(c) using latin1) c
from code_points
)
select
c,
weight_string(c collate latin1_bin) as bin_weight,
weight_string(c collate latin1_swedish_ci) as swedish_ci_weight
from characters
having bin_weight <> swedish_ci_weight;
```

Basically:

| Character | Code | Map to | Code |
|---|---|---|---|
| `a`–`z` | 0x61–0x7A | `A`–`Z` | 0x41–0x5A |
| `Å`, `å` | 0xC5, 0xE5 | - | 0x5B |
| `Ä`, `ä` | 0xC4, 0xE4 | - | 0x5C |
| `Æ`, `æ` | 0xC6, 0xE6 | - | 0x5C |
| `Ö`, `ö` | 0xD6 | - | 0x5D |
| `ÀÁÂÃ`, `àáâã` | 0xC0–0xC3, 0xE0–0xE3 | `A` | 0x41 |
| `Ç`, `ç` | 0xC7, 0xE7 | `C` | 0x43 |
| `Ð`, `ð` | 0xD0, 0xF0 | `D` | 0x44 |
| `ÈÉÊË`, `èéêë` | 0xC8–0xCB, 0xE8–0xEB | `E` | 0x45 |
| `ÌÍÎÏ`, `ìíîï` | 0xCC–0xCF, 0xEC–0xEF | `I` | 0x49 |
| `Ñ`, `ñ` | 0xD1, 0xF1 | `N` | 0x4E |
| `ÒÓÔÕ`, `òóôõ` | 0xD2–0xD5, 0xF2–0xF5 | `O` | 0x4F |
| `ÙÚÛ`, `ùúû` | 0xD9–0xDB, 0xF9–0xFB | `U` | 0x55 |
| `ÜÝ`, `üý` | 0xDC–0xDD, 0xFC–0xFD | `Y` | 0x59 |
| `ø` | 0xF8 | `Ø` | 0xD8 |
| `þ` | 0xFE | `Þ` | 0xDE |

**Describe alternatives you've considered:**

🤷

**Teachability, Documentation, Adoption, Migration Strategy:**

* Remove the "does not support `latin1_swedish_ci`" line from https://docs.pingcap.com/tidb/dev/migrate-from-mariadb/#character-set-and-collation
* Include `latin1_swedish_ci` support into https://docs.pingcap.com/tidb/dev/character-set-and-collation/

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.