Support `latin1_swedish_ci`
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Feature Request
**Is your feature request related to a problem? Please describe:**
[MySQL used `latin1_swedish_ci` as the default collation](https://dev.mysql.com/doc/refman/8.4/en/charset-we-sets.html#:~:text=The%20latin1%5Fswedish%5Fci%20collation%20is%20the%20default%20that%20probably%20is%20used%20by%20the%20majority%20of%20MySQL%20customers%2E) before v8.x. As such many historical databases upgraded from MySQL v5.x retained the `latin1_swedish_ci` columns, despite the developers or users not being Swedish🇸🇪 at all.
This imposes difficulty when migrating such databases into TiDB, because TiDB does not support `latin1_swedish_ci`. They have to change the collation in the downstream TiDB (the most compatible one is `utf8mb4_general_ci`), and this would require the user having to manually change the table schema downstream when using Lightning / DM, and also cross-charset comparison is buggy with sync-diff-inspector.
> Internal references: TCOC-1999, GTOC-6896, GTOC-7520, GTOC-8307
**Describe the feature you'd like:**
Actually implement the `latin1_swedish_ci` collation.
Running this in MySQL we can extract the WEIGHT_STRING mapping:
```sql
with recursive code_points as (
select 0 as c
union all
select c + 1
from code_points
where c < 255
), characters as (
select convert(char(c) using latin1) c
from code_points
)
select
c,
weight_string(c collate latin1_bin) as bin_weight,
weight_string(c collate latin1_swedish_ci) as swedish_ci_weight
from characters
having bin_weight <> swedish_ci_weight;
```
Basically:
| Character | Code | Map to | Code |
|---|---|---|---|
| `a`–`z` | 0x61–0x7A | `A`–`Z` | 0x41–0x5A |
| `Å`, `å` | 0xC5, 0xE5 | - | 0x5B |
| `Ä`, `ä` | 0xC4, 0xE4 | - | 0x5C |
| `Æ`, `æ` | 0xC6, 0xE6 | - | 0x5C |
| `Ö`, `ö` | 0xD6 | - | 0x5D |
| `ÀÁÂÃ`, `àáâã` | 0xC0–0xC3, 0xE0–0xE3 | `A` | 0x41 |
| `Ç`, `ç` | 0xC7, 0xE7 | `C` | 0x43 |
| `Ð`, `ð` | 0xD0, 0xF0 | `D` | 0x44 |
| `ÈÉÊË`, `èéêë` | 0xC8–0xCB, 0xE8–0xEB | `E` | 0x45 |
| `ÌÍÎÏ`, `ìíîï` | 0xCC–0xCF, 0xEC–0xEF | `I` | 0x49 |
| `Ñ`, `ñ` | 0xD1, 0xF1 | `N` | 0x4E |
| `ÒÓÔÕ`, `òóôõ` | 0xD2–0xD5, 0xF2–0xF5 | `O` | 0x4F |
| `ÙÚÛ`, `ùúû` | 0xD9–0xDB, 0xF9–0xFB | `U` | 0x55 |
| `ÜÝ`, `üý` | 0xDC–0xDD, 0xFC–0xFD | `Y` | 0x59 |
| `ø` | 0xF8 | `Ø` | 0xD8 |
| `þ` | 0xFE | `Þ` | 0xDE |
**Describe alternatives you've considered:**
🤷
**Teachability, Documentation, Adoption, Migration Strategy:**
* Remove the "does not support `latin1_swedish_ci`" line from https://docs.pingcap.com/tidb/dev/migrate-from-mariadb/#character-set-and-collation
* Include `latin1_swedish_ci` support into https://docs.pingcap.com/tidb/dev/character-set-and-collation/
Contributor guide
Assessment
This issue has not been assessed yet.