github / github/gh-ost

gh-ost crashed before cut-over phase

Open
#464 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

Hi all!

Yesterday I started a gh-ost process with this command:

```
gh-ost \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--throttle-control-replicas="production-readreplica" \
--max-lag-millis=1500 \
--host=production-readreplica \
--assume-master-host='production.xxxxxxxx:3306' \
--database="my_db_name" \
--table="activities" \
--verbose \
--alter="ADD IF NOT EXISTS reps_sets varchar(25) NOT NULL DEFAULT '0,0,0,0,0' COMMENT 'repetition based sets comma separated' AFTER set5, ADD IF NOT EXISTS time_sets varchar(25) NOT NULL DEFAULT '0,0,0,0,0' COMMENT 'time based sets comma separated' AFTER reps_sets, ADD IF NOT EXISTS time_based int NOT NULL DEFAULT '0' AFTER time_sets, CHANGE rest_period rest_period VARCHAR(25) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '30,30,30,30,30' COMMENT 'is rest between sets';" \
--assume-rbr \
--cut-over=default \
--exact-rowcount \
--concurrent-rowcount \
--initially-drop-ghost-table \
--default-retries=120 \
--postpone-cut-over-flag-file='/home/ubuntu/cut_over.flag' \
--hooks-path='/home/ubuntu/hooks' \
--conf='/home/ubuntu/gh-ost.cnf'
--allow-renamed-columns
```

content of gh-ost-on-before-cut-over-hook.sh:
```
#!/usr/bin/env bash

echo "Before cut-over started" >> /tmp/gh-ost.log

statement="UPDATE $GH_OST_GHOST_TABLE_NAME i SET i.reps_sets = CONCAT(i.set1,',',i.set2,',',i.set3,',',i.set4,',',i.set5),
rest_period = CONCAT(rest_period,',',rest_period,',',rest_period,',',rest_period,',',rest_period);"
echo "Issuing statement to update the ghost table before the cut-over: ${statement}" >> /tmp/gh-ost.log
exec mysql my_db_name -h production -uusername-ppassword -e "${statement}"
echo "Update finished" >> /tmp/gh-ost.log
```

So basically after the copy and before the switch, I wanted to do a big UPDATE.

I tried all first on a small table and both the copy, update and cutover (after deleting the flag file) went perfectly fine.

On the real table I issued a noop and everything went right, then a real --execute.
The process started correctly and it went on for 14h given the size of the table.

The problem is that this morning, for some reason, I woke up and gh-ost crashed.
I cannot find anymore the socket in /tmp or in any other place. The process just disappeared.

If I look in the db, I can see from the last line in the ghc table that the copy completed successfully:
```
240437 | 2017-08-04 01:17:05 | copy iteration 129266 at 1501809425 | Copy: 129265920/129265920 100.0%; Applied: 622862; Backlog: 0/1000; Time: 14h58m4s(total), 14h58m3s(copy); streamer: mysql-bin-changelog.000470:53163612; State: migrating; ETA: due
```
And my own /tmp/gh-ost.log doesn't contain the line:
`echo "Before cut-over started"`
so I can assume it never entered the before-cut-over phase.

Is there any way I can find more information about what happened?
Is there any way I can "resume" the process, instead of re-copy the table?

Thank you so much for the help and for this amazing tool!!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.