ERROR Timeout while waiting for events up to lock - after copy, well behind on binlog
- Dominant language
- Go
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 2h 31m
- Merged PRs (30d)
- 4
Description
I think this might be a little different than the other timeout, fail to cutover issues I'm seeing here.
I'm running gh-ost `1.0.47` on MySQL Aurora 1.16. Table I've been having problems with is about ~149G and about 5 million rows. Yeah, big table for so few rows. Had a gross text column.
This is the command I finally ran to successfully do the schema change:
```
gh-ost -allow-on-master -assume-rbr \
-critical-load Threads_running=600 -critical-load-hibernate-seconds 60 \
-database live_production -max-load Threads_running=400 -nice-ratio 0.1 \
-throttle-additional-flag-file ~/gh-ost-throttle_ghost \
--serve-socket-file=/tmp/gh-ost.sock \
-default-retries 1024 \
-chunk-size 5000 -ask-pass -user ${user} \
-table ${table} \
-host ${host} \
-alter 'ADD COLUMN `a` VARCHAR(3) DEFAULT NULL,
ADD COLUMN `b` INT DEFAULT NULL' \
-verbose -execute 2>&1 | tee gh-ost.log
```
Previous attempts were with less retries. After the copy was done, this was the binlog situation:
```
2019-02-12 04:13:17 INFO Waiting for events up to lock
Copy: 5242413/5242413 100.0%; Applied: 352; Backlog: 0/1000; Time: 2h40m10s(total), 2h35m47s(copy); streamer: mysql-bin-changelog.177290:164565360; State: migrating; ETA: due
2019-02-12 04:13:20 ERROR Timeout while waiting for events up to lock
2019-02-12 04:13:20 ERROR 2019-02-12 04:13:20 ERROR Timeout while waiting for events up to lock
```
```
mysql> show master status;
+----------------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+----------------------------+----------+--------------+------------------+-------------------+
| mysql-bin-changelog.177391 | 16275169 | | | |
+----------------------------+----------+--------------+------------------+-------------------+
```
So, as you can see, even though the backlog is 0, gh-ost is way behind on the binlog (177290 while 177391 was current). Almost 800 lock attempts latter, gh-ost finally caught up to current log and was able to successfully cut over.
Is this a bug or perhaps a parameters issue on my end? Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.