github / github/gh-ost

gh-ost idling on apply error

Open
#1,730 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

I was encountering problems when, at random times, gh-ost would just idle while backlogging was filling up, lag and heartbeatlag increasing, no new rows would get copied:
```
2026-06-26 09:05:27 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h7m0s(total), 3h6m59s(copy); streamer: dal13-sql21-bin.025288:95132715; Lag: 9236.81s, HeartbeatLag: 9236.81s, State: migrating; ETA: 157h7m35s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h7m30s(total), 3h7m29s(copy); streamer: dal13-sql21-bin.025288:120527768; Lag: 9266.80s, HeartbeatLag: 9266.81s, State: migrating; ETA: 157h32m48s
2026-06-26 09:05:57 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h7m30s(total), 3h7m29s(copy); streamer: dal13-sql21-bin.025288:120527768; Lag: 9266.80s, HeartbeatLag: 9266.81s, State: migrating; ETA: 157h32m48s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h8m0s(total), 3h7m59s(copy); streamer: dal13-sql21-bin.025288:149082002; Lag: 9296.80s, HeartbeatLag: 9296.81s, State: migrating; ETA: 157h58m1s
2026-06-26 09:06:27 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h8m0s(total), 3h7m59s(copy); streamer: dal13-sql21-bin.025288:149082002; Lag: 9296.80s, HeartbeatLag: 9296.81s, State: migrating; ETA: 157h58m1s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h8m30s(total), 3h8m29s(copy); streamer: dal13-sql21-bin.025288:172653166; Lag: 9326.71s, HeartbeatLag: 9326.81s, State: migrating; ETA: 158h23m13s
2026-06-26 09:06:57 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h8m30s(total), 3h8m29s(copy); streamer: dal13-sql21-bin.025288:172653166; Lag: 9326.71s, HeartbeatLag: 9326.81s, State: migrating; ETA: 158h23m13s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h9m0s(total), 3h8m59s(copy); streamer: dal13-sql21-bin.025288:194205311; Lag: 9356.80s, HeartbeatLag: 9356.81s, State: migrating; ETA: 158h48m26s
2026-06-26 09:07:27 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h9m0s(total), 3h8m59s(copy); streamer: dal13-sql21-bin.025288:194205311; Lag: 9356.80s, HeartbeatLag: 9356.81s, State: migrating; ETA: 158h48m26s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h9m30s(total), 3h9m29s(copy); streamer: dal13-sql21-bin.025288:217244680; Lag: 9386.80s, HeartbeatLag: 9386.81s, State: migrating; ETA: 159h13m38s
2026-06-26 09:07:57 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h9m30s(total), 3h9m29s(copy); streamer: dal13-sql21-bin.025288:217244680; Lag: 9386.80s, HeartbeatLag: 9386.81s, State: migrating; ETA: 159h13m38s []
```

I tried find out what is happening, but it wasn't throttled, couldn't find any issue for this idling.
So i enabled --verbose and found out that when processing the applied rows it encountered an error which made it just stay there:

```
2026-06-26 15:25:18 ERROR Error 1406 (22001): Data too long for column 'title' at row 1; query=
insert /* gh-ost.....
```

I think reporting that it encountered an error would be a good idea, and maybe aborting the copying, since there doesn't seem to be any method to resume the process.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the apply error with verbose logging and trace the applied-row processing path that stalls after MySQL error 1406. Done means the failure is clearly reported and the migration’s abort or resume behavior is established, with coverage for the data-too-long case.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, mysql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.