gh-ost not making progress on binlog position after being throttled?
- Dominant language
- Go
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 2h 31m
- Merged PRs (30d)
- 4
Description
Running the same migration described in #625 we found another puzzling issue. Looks like after reaching a certain point, gh-ost failed to keep reading events from the binary log.
Applied kept increasing and backlog was 3 on the last "good entry":
```
2018/08/10 11:53:15 binlogsyncer.go:573: [info] rotate to (master-bin.039662, 4)
2018/08/10 11:53:15 binlogsyncer.go:573: [info] rotate to (master-bin.039662, 4)
2018-08-10 11:53:15 INFO rotate to next log name: master-bin.039662
2018-08-10 11:53:15 INFO rotate to next log name: master-bin.039662
Copy: 18713759/20842379 89.8%; Applied: 6847; Backlog: 3/1000; Time: 7h51m0s(total), 7h50m8s(copy); streamer: master-bin.039662:10572413; State: throttled,
lag=6.250355s; ETA: 53m28s
```
after this, row copy continues but Applied doesn't move further than 6851, and the binlog position doesn't move past master-bin.039663:3282126:
```
2018/08/10 11:53:28 binlogsyncer.go:573: [info] rotate to (master-bin.039663, 4)
2018/08/10 11:53:28 binlogsyncer.go:573: [info] rotate to (master-bin.039663, 4)
2018-08-10 11:53:28 INFO rotate to next log name: master-bin.039663
2018-08-10 11:53:28 INFO rotate to next log name: master-bin.039663
Copy: 18714759/20842383 89.8%; Applied: 6851; Backlog: 0/1000; Time: 7h51m30s(total), 7h50m38s(copy); streamer: master-bin.039663:3282126; State: throttled
, lag=2.250244s; ETA: 53m30s
Copy: 18715759/20842383 89.8%; Applied: 6851; Backlog: 0/1000; Time: 7h52m0s(total), 7h51m8s(copy); streamer: master-bin.039663:3282126; State: throttled,
lag=18.050134s; ETA: 53m32s
Copy: 18715759/20842383 89.8%; Applied: 6851; Backlog: 0/1000; Time: 7h52m30s(total), 7h51m38s(copy); streamer: master-bin.039663:3282126; State: throttled, lag=2.650017s; ETA: 53m35s
Copy: 18716759/20842383 89.8%; Applied: 6851; Backlog: 0/1000; Time: 7h53m0s(total), 7h52m8s(copy); streamer: master-bin.039663:3282126; State: migrating; ETA: 53m37s
```
Eventually row copy finishes and as expected the cut-over fails due to timeout (as described in #625) , which we believe is due to gh-ost still being stuck in master-bin.039663:3282126.
I've manually inspected binary log at position mentioned (and entries around it) but nothing really stands out.
Any ideas why gh-ost could be getting stuck there?
Contributor guide
Assessment
This issue has not been assessed yet.