gh-ost crashed before cut-over phase
- Dominant language
- Go
- Stars
- 13.6k
- Forks
- 1.4k
- Avg merge
- 2h 31m
- Merged PRs (30d)
- 4
Description
Hi all!
Yesterday I started a gh-ost process with this command:
```
gh-ost \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--throttle-control-replicas="production-readreplica" \
--max-lag-millis=1500 \
--host=production-readreplica \
--assume-master-host='production.xxxxxxxx:3306' \
--database="my_db_name" \
--table="activities" \
--verbose \
--alter="ADD IF NOT EXISTS reps_sets varchar(25) NOT NULL DEFAULT '0,0,0,0,0' COMMENT 'repetition based sets comma separated' AFTER set5, ADD IF NOT EXISTS time_sets varchar(25) NOT NULL DEFAULT '0,0,0,0,0' COMMENT 'time based sets comma separated' AFTER reps_sets, ADD IF NOT EXISTS time_based int NOT NULL DEFAULT '0' AFTER time_sets, CHANGE rest_period rest_period VARCHAR(25) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '30,30,30,30,30' COMMENT 'is rest between sets';" \
--assume-rbr \
--cut-over=default \
--exact-rowcount \
--concurrent-rowcount \
--initially-drop-ghost-table \
--default-retries=120 \
--postpone-cut-over-flag-file='/home/ubuntu/cut_over.flag' \
--hooks-path='/home/ubuntu/hooks' \
--conf='/home/ubuntu/gh-ost.cnf'
--allow-renamed-columns
```
content of gh-ost-on-before-cut-over-hook.sh:
```
#!/usr/bin/env bash
echo "Before cut-over started" >> /tmp/gh-ost.log
statement="UPDATE $GH_OST_GHOST_TABLE_NAME i SET i.reps_sets = CONCAT(i.set1,',',i.set2,',',i.set3,',',i.set4,',',i.set5),
rest_period = CONCAT(rest_period,',',rest_period,',',rest_period,',',rest_period,',',rest_period);"
echo "Issuing statement to update the ghost table before the cut-over: ${statement}" >> /tmp/gh-ost.log
exec mysql my_db_name -h production -uusername-ppassword -e "${statement}"
echo "Update finished" >> /tmp/gh-ost.log
```
So basically after the copy and before the switch, I wanted to do a big UPDATE.
I tried all first on a small table and both the copy, update and cutover (after deleting the flag file) went perfectly fine.
On the real table I issued a noop and everything went right, then a real --execute.
The process started correctly and it went on for 14h given the size of the table.
The problem is that this morning, for some reason, I woke up and gh-ost crashed.
I cannot find anymore the socket in /tmp or in any other place. The process just disappeared.
If I look in the db, I can see from the last line in the ghc table that the copy completed successfully:
```
240437 | 2017-08-04 01:17:05 | copy iteration 129266 at 1501809425 | Copy: 129265920/129265920 100.0%; Applied: 622862; Backlog: 0/1000; Time: 14h58m4s(total), 14h58m3s(copy); streamer: mysql-bin-changelog.000470:53163612; State: migrating; ETA: due
```
And my own /tmp/gh-ost.log doesn't contain the line:
`echo "Before cut-over started"`
so I can assume it never entered the before-cut-over phase.
Is there any way I can find more information about what happened?
Is there any way I can "resume" the process, instead of re-copy the table?
Thank you so much for the help and for this amazing tool!!
Contributor guide
Assessment
This issue has not been assessed yet.