Row counts not matching at cutover and then cutover continually fails and loops
- Dominant language
- Go
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 2h 31m
- Merged PRs (30d)
- 4
Description
We've been trying to migrate a fairly large table, it has about 100 million rows and its about 150 gb. We've done two attempts at adding the columns and in both cases when we got to the postponed cutover I manually ran count(1) queries on the original table and the _gho table and the row counts mismatched the first time by 300k rows and the second time by 1M rows. We're able to stop writes to this table and so we still attempted the cutover process to see if it would catch things up, but it seemed to loop with the following error:
```
2017-08-25 13:26:47 INFO Looking for magic cut-over table
2017-08-25 13:26:47 INFO Will now proceed to drop magic table and unlock tables
2017-08-25 13:26:47 INFO Dropping magic cut-over table
2017-08-25 13:26:47 INFO Releasing lock from `db`.`table_a`, `db`.`_table_a_del`
2017-08-25 13:26:47 INFO Tables unlocked
2017-08-25 13:26:48 INFO Grabbing voluntary lock: gh-ost.125413.lock
2017-08-25 13:26:48 INFO Setting LOCK timeout as 6 seconds
2017-08-25 13:26:48 INFO Looking for magic cut-over table
2017-08-25 13:26:48 INFO Creating magic cut-over table `db`.`_table_a_del`
2017-08-25 13:26:48 INFO Magic cut-over table created
2017-08-25 13:26:48 INFO Locking `db`.`table_a`, `db`.`_table_a_del`
2017-08-25 13:26:48 INFO Tables locked
2017-08-25 13:26:48 INFO Session locking original & magic tables is 125413
2017-08-25 13:26:48 INFO Writing changelog state: AllEventsUpToLockProcessed:1503667608367587880
2017-08-25 13:26:48 INFO Waiting for events up to lock
2017-08-25 13:26:51 ERROR Timeout while waiting for events up to lock
2017-08-25 13:26:51 ERROR 2017-08-25 13:26:51 ERROR Timeout while waiting for events up to lock
2017-08-25 13:26:51 INFO Looking for magic cut-over table
2017-08-25 13:26:51 INFO Will now proceed to drop magic table and unlock tables
2017-08-25 13:26:51 INFO Dropping magic cut-over table
2017-08-25 13:26:51 INFO Releasing lock from `db`.`table_a`, `db`.`_table_a_del`
2017-08-25 13:26:51 INFO Tables unlocked
```
It repeats this output over and over. The first time around it eventually failed due to the default 60 retries, but the second time we increased that so it retried longer. It wasn't making any progress on catching up the missing rows.
We're running MySQL 5.7.17 in Amazon RDS.
I currently still have the migration attempt running with the cutover postponed, so if there are things to try out or information gathered that can be done without restarting it that would be great, since it took 88 hours to get to this point.
We run ghost with the following params, and have had alot of success until this table.
```
--host=${slave_host} \
--table="${table_name}" \
--alter="${alter_statement}" \
--assume-master-host=${master_host} \
--database="${database_name}" \
--throttle-control-replicas=${slave_host} \
--assume-rbr \
--initially-drop-ghost-table \
--initially-drop-socket-file \
--max-load=Threads_running=15 \
--max-lag-millis=1000 \
--chunk-size=1000 \
--nice-ratio=2.5 \
--verbose \
--cut-over=default \
--cut-over-lock-timeout-seconds=3 \
--default-retries=1000000 \
--panic-flag-file=/tmp/${script_identifier}-gh-ost.panic.flag \
--serve-socket-file=/tmp/${script_identifier}-gh-ost.sock \
--throttle-flag-file=/tmp/${script_identifier}-gh-ost.throttle \
--postpone-cut-over-flag-file=/tmp/${script_identifier}-gh-ost.postpone \
--execute
```
Contributor guide
Assessment
This issue has not been assessed yet.