Migration over 100% and staying in migrating state. Never in postpone state.
- Dominant language
- Go
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 2h 31m
- Merged PRs (30d)
- 4
Description
We are unable to run the unpostpone command.
The migration says over 100% and when we do the `unpostpone` command, if says `You may only invoke this when gh-ost is actively postponing migration. At this time it is not.`. The state is also saying `migrating` when it should say `postpone`
Any idea how to resolve this?
Command to run:
```
gh-ost
--max-load=Threads_running=3000,threads_connected=20000
--critical-load=Threads_running=3000
--chunk-size=1000
--max-lag-millis=1500
--allow-on-master
--verbose
--assume-rbr
--allow-master-master
--cut-over=default
--exact-rowcount
--concurrent-rowcount
--default-retries=20
--panic-flag-file=/tmp/ghost.panic.flag
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag
--user=
--password=
--host=
--database=
--table=orders
--dml-batch-size=100
--alter=';'
--execute
```
Here are the logs from gh-ost:
```2019-07-20 03:46:47 INFO rotate to next log from mysql-bin-changelog.411212:134318219 to mysql-bin-changelog.411212
2019-07-20 03:46:47 INFO rotate to next log from mysql-bin-changelog.411212:0 to mysql-bin-changelog.411212
Copy: 33559300/30109847 111.5%; Applied: 442632; Backlog: 0/1000; Time: 6h51m30s(total), 6h51m28s(copy); streamer: mysql-bin-changelog.411212:7928536; State: migrating; ETA: due
Copy: 33584000/30109953 111.5%; Applied: 443032; Backlog: 15/1000; Time: 6h52m0s(total), 6h51m58s(copy); streamer: mysql-bin-changelog.411212:113314793; State: migrating; ETA: due
[2019/07/20 03:47:24] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411213, 4)
[2019/07/20 03:47:24] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411213, 4)
2019-07-20 03:47:24 INFO rotate to next log from mysql-bin-changelog.411213:134278535 to mysql-bin-changelog.411213
2019-07-20 03:47:24 INFO rotate to next log from mysql-bin-changelog.411213:0 to mysql-bin-changelog.411213
Copy: 33614500/30110056 111.6%; Applied: 443431; Backlog: 0/1000; Time: 6h52m30s(total), 6h52m28s(copy); streamer: mysql-bin-changelog.411213:95973538; State: migrating; ETA: due
[2019/07/20 03:47:58] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411214, 4)
[2019/07/20 03:47:58] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411214, 4)
2019-07-20 03:47:58 INFO rotate to next log from mysql-bin-changelog.411214:134292681 to mysql-bin-changelog.411214
2019-07-20 03:47:58 INFO rotate to next log from mysql-bin-changelog.411214:0 to mysql-bin-changelog.411214
Copy: 33641700/30110158 111.7%; Applied: 443810; Backlog: 0/1000; Time: 6h53m0s(total), 6h52m58s(copy); streamer: mysql-bin-changelog.411214:71714977; State: migrating; ETA: due
[2019/07/20 03:48:36] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411215, 4)
[2019/07/20 03:48:36] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411215, 4)
2019-07-20 03:48:36 INFO rotate to next log from mysql-bin-changelog.411215:134317693 to mysql-bin-changelog.411215
2019-07-20 03:48:36 INFO rotate to next log from mysql-bin-changelog.411215:0 to mysql-bin-changelog.411215
Copy: 33671900/30110274 111.8%; Applied: 444268; Backlog: 0/1000; Time: 6h53m30s(total), 6h53m28s(copy); streamer: mysql-bin-changelog.411215:50180359; State: migrating; ETA: due
[2019/07/20 03:49:14] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411216, 4)
[2019/07/20 03:49:14] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411216, 4)
2019-07-20 03:49:14 INFO rotate to next log from mysql-bin-changelog.411216:134339835 to mysql-bin-changelog.411216
2019-07-20 03:49:14 INFO rotate to next log from mysql-bin-changelog.411216:0 to mysql-bin-changelog.411216
Copy: 33697000/30110371 111.9%; Applied: 444643; Backlog: 0/1000; Time: 6h54m0s(total), 6h53m58s(copy); streamer: mysql-bin-changelog.411216:16571288; State: migrating; ETA: due
Copy: 33724200/30110463 112.0%; Applied: 444986; Backlog: 0/1000; Time: 6h54m30s(total), 6h54m28s(copy); streamer: mysql-bin-changelog.411216:126988305; State: migrating; ETA: due
[2019/07/20 03:49:51] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411217, 4)
[2019/07/20 03:49:51] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411217, 4)
2019-07-20 03:49:51 INFO rotate to next log from mysql-bin-changelog.411217:134320044 to mysql-bin-changelog.411217
2019-07-20 03:49:51 INFO rotate to next log from mysql-bin-changelog.411217:0 to mysql-bin-changelog.411217
[2019/07/20 03:50:00] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411218, 4)
[2019/07/20 03:50:00] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411218, 4)
2019-07-20 03:50:00 INFO rotate to next log from mysql-bin-changelog.411218:31034577 to mysql-bin-changelog.411218
2019-07-20 03:50:00 INFO rotate to next log from mysql-bin-changelog.411218:0 to mysql-bin-changelog.411218
Copy: 33754100/30110555 112.1%; Applied: 445335; Backlog: 0/1000; Time: 6h55m0s(total), 6h54m58s(copy); streamer: mysql-bin-changelog.411218:71285399; State: migrating; ETA: due
[2019/07/20 03:50:39] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411219, 4)
[2019/07/20 03:50:39] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411219, 4)
2019-07-20 03:50:39 INFO rotate to next log from mysql-bin-changelog.411219:134264206 to mysql-bin-changelog.411219
2019-07-20 03:50:39 INFO rotate to next log from mysql-bin-changelog.411219:0 to mysql-bin-changelog.411219
Copy: 33780700/30110659 112.2%; Applied: 445738; Backlog: 0/1000; Time: 6h55m30s(total), 6h55m28s(copy); streamer: mysql-bin-changelog.411219:32264928; State: migrating; ETA: due
Copy: 33809100/30110793 112.3%; Applied: 446249; Backlog: 0/1000; Time: 6h56m0s(total), 6h55m58s(copy); streamer: mysql-bin-changelog.411219:131014331; State: migrating; ETA: due
[2019/07/20 03:51:20] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411220, 4)
[2019/07/20 03:51:20] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411220, 4)
2019-07-20 03:51:20 INFO rotate to next log from mysql-bin-changelog.411220:134265429 to mysql-bin-changelog.411220
2019-07-20 03:51:20 INFO rotate to next log from mysql-bin-changelog.411220:0 to mysql-bin-changelog.411220````
Contributor guide
Assessment
This issue has not been assessed yet.