github / github/gh-ost

Gh-ost backlog 1000/1000 can't make any progress.

Open
#761 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

Hi I work at Lime and we use gh-ost and like it a lot, but when we are running a migration on a table with a lot of writes, it can not complete. We have ran gh-ost on various tables with success but when there are a high number of writes we can't get it to complete.

The backlog gets stuck at 1000 and nothing gets copied.

We are running gh-ost in a job in Kubernetes.

Running the following:
```
gh-ost
--max-load=Threads_running=3000,threads_connected=20000
--critical-load=Threads_running=3000
--chunk-size=1000
--max-lag-millis=1500
--allow-on-master
--verbose
--assume-rbr
--allow-master-master
--cut-over=default
--exact-rowcount
--concurrent-rowcount
--default-retries=20
--panic-flag-file=/tmp/ghost.panic.flag
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag
--user=
--password=
--host=
--database=
--table=bikes
--dml-batch-size=100
--alter='ALTER TABLE bikes ADD COLUMN location point;'
--execute
```

Output:

```
Copy: 0/714541 0.0%; Applied: 49550; Backlog: 1000/1000; Time: 1m20s(total), 1m19s(copy); streamer: mysql-bin-changelog.374920:125825751; State: migrating; ETA: N/A
[2019/07/03 19:18:26] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.374921, 4)
[2019/07/03 19:18:26] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.374921, 4)
Copy: 0/714541 0.0%; Applied: 52700; Backlog: 1000/1000; Time: 1m25s(total), 1m24s(copy); streamer: mysql-bin-changelog.374920:133856640; State: migrating; ETA: N/A
2019-07-03 19:18:27 INFO rotate to next log from mysql-bin-changelog.374921:134286986 to mysql-bin-changelog.374921
2019-07-03 19:18:27 INFO rotate to next log from mysql-bin-changelog.374921:0 to mysql-bin-changelog.374921
Copy: 0/714541 0.0%; Applied: 55890; Backlog: 1000/1000; Time: 1m30s(total), 1m29s(copy); streamer: mysql-bin-changelog.374921:7742880; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 59080; Backlog: 1000/1000; Time: 1m35s(total), 1m34s(copy); streamer: mysql-bin-changelog.374921:15603076; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 62270; Backlog: 1000/1000; Time: 1m40s(total), 1m39s(copy); streamer: mysql-bin-changelog.374921:23465989; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 65420; Backlog: 1000/1000; Time: 1m45s(total), 1m44s(copy); streamer: mysql-bin-changelog.374921:31669793; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 68570; Backlog: 1000/1000; Time: 1m50s(total), 1m49s(copy); streamer: mysql-bin-changelog.374921:41022300; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 71750; Backlog: 1000/1000; Time: 1m55s(total), 1m54s(copy); streamer: mysql-bin-changelog.374921:51120678; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 74920; Backlog: 1000/1000; Time: 2m0s(total), 1m59s(copy); streamer: mysql-bin-changelog.374921:58903179; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 78100; Backlog: 1000/1000; Time: 2m5s(total), 2m4s(copy); streamer: mysql-bin-changelog.374921:66963586; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 81310; Backlog: 1000/1000; Time: 2m10s(total), 2m9s(copy); streamer: mysql-bin-changelog.374921:74901698; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 84460; Backlog: 1000/1000; Time: 2m15s(total), 2m14s(copy); streamer: mysql-bin-changelog.374921:82858858; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 87590; Backlog: 1000/1000; Time: 2m20s(total), 2m19s(copy); streamer: mysql-bin-changelog.374921:90644007; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 90720; Backlog: 1000/1000; Time: 2m25s(total), 2m24s(copy); streamer: mysql-bin-changelog.374921:98630590; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 93850; Backlog: 1000/1000; Time: 2m30s(total), 2m29s(copy); streamer: mysql-bin-changelog.374921:106711984; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 96990; Backlog: 1000/1000; Time: 2m35s(total), 2m34s(copy); streamer: mysql-bin-changelog.374921:114554349; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 100070; Backlog: 1000/1000; Time: 2m40s(total), 2m39s(copy); streamer: mysql-bin-changelog.374921:122465210; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 103240; Backlog: 1000/1000; Time: 2m45s(total), 2m44s(copy); streamer: mysql-bin-changelog.374921:130295653; State: migrating; ETA: N/A
[2019/07/03 19:19:48] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.374922, 4)
[2019/07/03 19:19:48] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.374922, 4)
2019-07-03 19:19:49 INFO rotate to next log from mysql-bin-changelog.374922:134270583 to mysql-bin-changelog.374922
2019-07-03 19:19:49 INFO rotate to next log from mysql-bin-changelog.374922:0 to mysql-bin-changelog.374922
Copy: 0/714541 0.0%; Applied: 106380; Backlog: 1000/1000; Time: 2m50s(total), 2m49s(copy); streamer: mysql-bin-changelog.374922:4030170; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 109480; Backlog: 1000/1000; Time: 2m55s(total), 2m54s(copy); streamer: mysql-bin-changelog.374922:11866933; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 112620; Backlog: 1000/1000; Time: 3m0s(total), 2m59s(copy); streamer: mysql-bin-changelog.374922:19908039; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 131450; Backlog: 1000/1000; Time: 3m30s(total), 3m29s(copy); streamer: mysql-bin-changelog.374922:68951154; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 150120; Backlog: 1000/1000; Time: 4m0s(total), 3m59s(copy); streamer: mysql-bin-changelog.374922:127018597; State: migrating; ETA: N/A
[2019/07/03 19:21:05] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.374923, 4)
[2019/07/03 19:21:05] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.374923, 4)
2019-07-03 19:21:06 INFO rotate to next log from mysql-bin-changelog.374923:134291901 to mysql-bin-changelog.374923
2019-07-03 19:21:06 INFO rotate to next log from mysql-bin-changelog.374923:0 to mysql-bin-changelog.374923
Copy: 0/714541 0.0%; Applied: 169190; Backlog: 993/1000; Time: 4m30s(total), 4m29s(copy); streamer: mysql-bin-changelog.374923:53001257; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 188340; Backlog: 993/1000; Time: 5m0s(total), 4m59s(copy); streamer: mysql-bin-changelog.374923:115111365; State: migrating; ETA: N/A
[2019/07/03 19:22:13] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.374924, 4)
[2019/07/03 19:22:13] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.374924, 4)
2019-07-03 19:22:14 INFO rotate to next log from mysql-bin-changelog.374924:134307761 to mysql-bin-changelog.374924
2019-07-03 19:22:14 INFO rotate to next log from mysql-bin-changelog.374924:0 to mysql-bin-changelog.374924
Copy: 0/714541 0.0%; Applied: 207380; Backlog: 1000/1000; Time: 5m30s(total), 5m29s(copy); streamer: mysql-bin-changelog.374924:42299534; State: migrating; ETA: N/A
[2019/07/03 19:22:44] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.374925, 4)
[2019/07/03 19:22:44] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.374925, 4)
2019-07-03 19:22:45 INFO rotate to next log from mysql-bin-changelog.374925:64729630 to mysql-bin-changelog.374925
2019-07-03 19:22:45 INFO rotate to next log from mysql-bin-changelog.374925:0 to mysql-bin-changelog.374925
Copy: 0/714541 0.0%; Applied: 226520; Backlog: 1000/1000; Time: 6m0s(total), 5m59s(copy); streamer: mysql-bin-changelog.374925:27583532; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 245510; Backlog: 1000/1000; Time: 6m30s(total), 6m29s(copy); streamer: mysql-bin-changelog.374925:78625914; State: migrating; ETA: N/A
Copy: 0/714541 0.0%; Applied: 264500; Backlog: 1000/1000; Time: 7m0s(total), 6m59s(copy); streamer: mysql-bin-changelog.374925:130436948; State: migrating; ETA: N/A
[2019/07/03 19:24:03] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.374926, 4)
[2019/07/03 19:24:03] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.374926, 4)
```

Gh-ost version: 1.0.48

SQL:
`ALTER TABLE bikes ALTER TABLE bikes ADD COLUMN location point;`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.