github / github/gh-ost

Cutover not succeeding

Open
#687 23 comments 3 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

When I try to run a migration to add an index on a relatively small table, the cutover never succeeds. I've run it with verbose output, and get the following:
```
2018-12-06 14:43:42 INFO Locking `COMPANY_prod`.`retailers`, `COMPANY_prod`.`_retailers_del`
Copy: 91156/91156 100.0%; Applied: 2; Backlog: 0/1000; Time: 1m35s(total), 39s(copy); streamer: mysql-bin-changelog.002933:13183392; State: migrating; ETA: due
2018-12-06 14:43:48 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction
2018-12-06 14:43:48 INFO Looking for magic cut-over table
2018-12-06 14:43:48 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction
2018-12-06 14:43:48 INFO Dropping magic cut-over table
2018-12-06 14:43:48 INFO Dropping table `COMPANY_prod`.`_retailers_del`
2018-12-06 14:43:48 INFO Table dropped
2018-12-06 14:43:49 INFO Grabbing voluntary lock: gh-ost.12668.lock
2018-12-06 14:43:49 INFO Setting LOCK timeout as 6 seconds
2018-12-06 14:43:49 INFO Looking for magic cut-over table
2018-12-06 14:43:49 INFO Creating magic cut-over table `COMPANY_prod`.`_retailers_del`
2018-12-06 14:43:50 INFO Magic cut-over table created
2018-12-06 14:43:50 INFO Locking `COMPANY_prod`.`retailers`, `COMPANY_prod`.`_retailers_del`
Copy: 91156/91156 100.0%; Applied: 4; Backlog: 0/1000; Time: 1m40s(total), 39s(copy); streamer: mysql-bin-changelog.002933:13689223; State: migrating; ETA: due
2018-12-06 14:43:56 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction
2018-12-06 14:43:56 INFO Looking for magic cut-over table
2018-12-06 14:43:56 INFO Dropping magic cut-over table
2018-12-06 14:43:56 INFO Dropping table `COMPANY_prod`.`_retailers_del`
2018-12-06 14:43:56 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction
2018-12-06 14:43:56 INFO Table dropped
Copy: 91156/91156 100.0%; Applied: 4; Backlog: 0/1000; Time: 1m45s(total), 39s(copy); streamer: mysql-bin-changelog.002933:14037660; State: migrating; ETA: due
2018-12-06 14:43:57 INFO Grabbing voluntary lock: gh-ost.12680.lock
2018-12-06 14:43:57 INFO Setting LOCK timeout as 6 seconds
2018-12-06 14:43:57 INFO Looking for magic cut-over table
2018-12-06 14:43:57 INFO Creating magic cut-over table `COMPANY_prod`.`_retailers_del`
2018-12-06 14:43:57 INFO Magic cut-over table created
2018-12-06 14:43:57 INFO Locking `COMPANY_prod`.`retailers`, `COMPANY_prod`.`_retailers_del`
Copy: 91156/91156 100.0%; Applied: 4; Backlog: 0/1000; Time: 1m50s(total), 39s(copy); streamer: mysql-bin-changelog.002933:14381575; State: migrating; ETA: due
2018-12-06 14:44:03 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction
2018-12-06 14:44:03 INFO Looking for magic cut-over table
2018-12-06 14:44:03 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction
2018-12-06 14:44:03 INFO Dropping magic cut-over table
2018-12-06 14:44:03 INFO Dropping table `COMPANY_prod`.`_retailers_del`
2018-12-06 14:44:03 INFO Table dropped
2018-12-06 14:44:05 INFO Grabbing voluntary lock: gh-ost.12685.lock
2018-12-06 14:44:05 INFO Setting LOCK timeout as 6 seconds
2018-12-06 14:44:05 INFO Looking for magic cut-over table
2018-12-06 14:44:05 INFO Creating magic cut-over table `COMPANY_prod`.`_retailers_del`
2018-12-06 14:44:05 INFO Magic cut-over table created
2018-12-06 14:44:05 INFO Locking `COMPANY_prod`.`retailers`, `COMPANY_prod`.`_retailers_del`
Copy: 91156/91156 100.0%; Applied: 4; Backlog: 0/1000; Time: 1m55s(total), 39s(copy); streamer: mysql-bin-changelog.002933:15039450; State: migrating; ETA: due
```

It will repeat this ad infinitum, until I kill it after 3-5 minutes. We _are_ using Aurora, but have RBR set and `run-on-master`. We're able to run other migrations, except for this (relatively) small one, so I'm trying to figure out what possibly could cause this so my debugging can be a bit more guided.

So far, love the tool and the fact I don't have to worry about cleanup after, it's been great for us. Just need to figure out this occasional issue!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.