gh-ost idling on apply error
- Dominant language
- Go
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 2h 31m
- Merged PRs (30d)
- 4
Description
I was encountering problems when, at random times, gh-ost would just idle while backlogging was filling up, lag and heartbeatlag increasing, no new rows would get copied:
```
2026-06-26 09:05:27 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h7m0s(total), 3h6m59s(copy); streamer: dal13-sql21-bin.025288:95132715; Lag: 9236.81s, HeartbeatLag: 9236.81s, State: migrating; ETA: 157h7m35s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h7m30s(total), 3h7m29s(copy); streamer: dal13-sql21-bin.025288:120527768; Lag: 9266.80s, HeartbeatLag: 9266.81s, State: migrating; ETA: 157h32m48s
2026-06-26 09:05:57 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h7m30s(total), 3h7m29s(copy); streamer: dal13-sql21-bin.025288:120527768; Lag: 9266.80s, HeartbeatLag: 9266.81s, State: migrating; ETA: 157h32m48s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h8m0s(total), 3h7m59s(copy); streamer: dal13-sql21-bin.025288:149082002; Lag: 9296.80s, HeartbeatLag: 9296.81s, State: migrating; ETA: 157h58m1s
2026-06-26 09:06:27 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h8m0s(total), 3h7m59s(copy); streamer: dal13-sql21-bin.025288:149082002; Lag: 9296.80s, HeartbeatLag: 9296.81s, State: migrating; ETA: 157h58m1s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h8m30s(total), 3h8m29s(copy); streamer: dal13-sql21-bin.025288:172653166; Lag: 9326.71s, HeartbeatLag: 9326.81s, State: migrating; ETA: 158h23m13s
2026-06-26 09:06:57 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h8m30s(total), 3h8m29s(copy); streamer: dal13-sql21-bin.025288:172653166; Lag: 9326.71s, HeartbeatLag: 9326.81s, State: migrating; ETA: 158h23m13s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h9m0s(total), 3h8m59s(copy); streamer: dal13-sql21-bin.025288:194205311; Lag: 9356.80s, HeartbeatLag: 9356.81s, State: migrating; ETA: 158h48m26s
2026-06-26 09:07:27 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h9m0s(total), 3h8m59s(copy); streamer: dal13-sql21-bin.025288:194205311; Lag: 9356.80s, HeartbeatLag: 9356.81s, State: migrating; ETA: 158h48m26s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h9m30s(total), 3h9m29s(copy); streamer: dal13-sql21-bin.025288:217244680; Lag: 9386.80s, HeartbeatLag: 9386.81s, State: migrating; ETA: 159h13m38s
2026-06-26 09:07:57 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h9m30s(total), 3h9m29s(copy); streamer: dal13-sql21-bin.025288:217244680; Lag: 9386.80s, HeartbeatLag: 9386.81s, State: migrating; ETA: 159h13m38s []
```
I tried find out what is happening, but it wasn't throttled, couldn't find any issue for this idling.
So i enabled --verbose and found out that when processing the applied rows it encountered an error which made it just stay there:
```
2026-06-26 15:25:18 ERROR Error 1406 (22001): Data too long for column 'title' at row 1; query=
insert /* gh-ost.....
```
I think reporting that it encountered an error would be a good idea, and maybe aborting the copying, since there doesn't seem to be any method to resume the process.
Contributor guide
Research direction
Start by reproducing the apply error with verbose logging and trace the applied-row processing path that stalls after MySQL error 1406. Done means the failure is clearly reported and the migration’s abort or resume behavior is established, with coverage for the data-too-long case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, mysql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100