github / github/gh-ost

Errors When Applying Binlog for latin1 Character Set

Open
#814 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

We have a number of tables in our database that unfortunately use the `latin1` character set. We need to migrate these to `utf8mb4` and are hoping to use gh-ost to facilitate this. In doing some testing, I'm running into a problem when changes from the binlog are being applied. I can reliably produce the problem as follows:

1. SQL setup
```sql
create table test_charset(id int not null primary key auto_increment, answer text) default charset=latin1; # This is simpler than our real table but illustrates the problem
insert into test_charset(answer) values('Here’s an apostrophe'); # note that that's a "fancy" apostrophe
```
2. gh-ost command (the strange ports are for Docker containers running locally):
```bash
bin/gh-ost --user="root" --password="" --host="127.0.0.1" --port=5506 --database=loyalty --table="test_charset" --verbose --alter="modify column answer text charset utf8mb4" --exact-rowcount --assume-rbr --postpone-cut-over-flag-file="/tmp/gh-ost-cutover.flag" --max-lag-millis=7500 --assume-master-host="127.0.0.1:4406" --execute
```

The migration proceeds as expected when copying rows and then waits since the cutover flag file is present. If I then insert the same row again, gh-ost crashes:
```sql
insert into test_charset(answer) values('Here’s an apostrophe');
```

The gh-ost output is attached. It seems that the value of the `answer` column is coming through as `[]uint8` instead of `string` so the character set conversion in [types.go/convertArg](https://github.com/github/gh-ost/blob/master/go/sql/types.go#L42) isn't happening. The "fancy" apostrophe is valid in the `latin1` character set.

Any advice is appreciated.

[gh-ost.log](https://github.com/github/gh-ost/files/4042176/gh-ost.log)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.