Migrating 100%, but shows State: postponing cut-over; ETA: due

Hi, I'm new to this. I try to use this greate tool for modifying table structure. When migrating completed 100%, but continuously pop up:

> Copy: 22102848/22102848 100.0%; Applied: 110; Backlog: 0/1000; Time: 3h22m0s(total), 3h6m12s(copy); streamer: mysql-bin.000018:912494008; State: postponing cut-over; ETA: due

and my command is this:

> ./gh-ost \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--max-lag-millis=1500 \
--user="xxxx" \
--password="xxxx" \
--host=xxxx \
--port=4223 \
--database="test" \
--table="t1" \
--verbose \
--alter="ADD COLUMN isBookmarkedThree TINYINT(1) NOT NULL DEFAULT 0" \
--allow-master-master \
--cut-over=default \
--exact-rowcount \
--concurrent-rowcount \
--default-retries=120 \
--panic-flag-file=./ghost.panic.flag \
--postpone-cut-over-flag-file=./ghost.postpone.flag \
-aliyun-rds \
--allow-on-master \
--execute

I try to delete ghost.postpone.flag file, it kept popping up:

> 2019-12-06 19:25:39 INFO Grabbing voluntary lock: gh-ost.319743.lock
2019-12-06 19:25:39 INFO Setting LOCK timeout as 6 seconds
2019-12-06 19:25:39 INFO Looking for magic cut-over table
2019-12-06 19:25:39 INFO Creating magic cut-over table `test`.`_t1_del`
2019-12-06 19:25:39 INFO Magic cut-over table created
2019-12-06 19:25:39 INFO Locking `test`.`t1`, `test`.`_t1_del`
2019-12-06 19:25:39 INFO Tables locked
2019-12-06 19:25:39 INFO Session locking original & magic tables is 319743
2019-12-06 19:25:39 INFO Writing changelog state: AllEventsUpToLockProcessed:1575631539838703726
2019-12-06 19:25:39 INFO Waiting for events up to lock
2019-12-06 19:25:42 ERROR Timeout while waiting for events up to lock
2019-12-06 19:25:42 ERROR 2019-12-06 19:25:42 ERROR Timeout while waiting for events up to lock
2019-12-06 19:25:42 INFO Looking for magic cut-over table
2019-12-06 19:25:42 INFO Will now proceed to drop magic table and unlock tables
2019-12-06 19:25:42 INFO Dropping magic cut-over table
2019-12-06 19:25:42 INFO Releasing lock from `test`.`t1`, `test`.`_t1_del`
2019-12-06 19:25:42 INFO Tables unlocked

serveral minutes later, ends like:

> 2019-12-06 19:27:39 ERROR Timeout while waiting for events up to lock
2019-12-06 19:27:39 ERROR 2019-12-06 19:27:39 ERROR Timeout while waiting for events up to lock
2019-12-06 19:27:39 INFO Looking for magic cut-over table
2019-12-06 19:27:39 INFO Will now proceed to drop magic table and unlock tables
2019-12-06 19:27:39 INFO Dropping magic cut-over table
2019-12-06 19:27:39 INFO Removing socket file: /tmp/gh-ost.test.t1.sock
2019-12-06 19:27:39 FATAL 2019-12-06 19:27:39 ERROR Timeout while waiting for events up to lock

Is there anything I missed? Thanks

Start with the reported --postpone-cut-over-flag-file and --cut-over=default settings, then trace the cut-over sequence represented by the log messages. Investigate the repeated “Timeout while waiting for events up to lock” failure in the MySQL migration context. Done means identifying the cause and documenting the required configuration or behavior for a successful cut-over.

Scritto dal modello di indicizzazione a partire dal testo della issue.