github / github/gh-ost

gh-ost crashed before cut-over phase

Aperta
#464 5 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Go
Stelle
13.6k
Fork
1.4k
Merge medio
2h 31m
PR unite (30g)
4

Descrizione

Hi all!

Yesterday I started a gh-ost process with this command:

```
gh-ost \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--throttle-control-replicas="production-readreplica" \
--max-lag-millis=1500 \
--host=production-readreplica \
--assume-master-host='production.xxxxxxxx:3306' \
--database="my_db_name" \
--table="activities" \
--verbose \
--alter="ADD IF NOT EXISTS reps_sets varchar(25) NOT NULL DEFAULT '0,0,0,0,0' COMMENT 'repetition based sets comma separated' AFTER set5, ADD IF NOT EXISTS time_sets varchar(25) NOT NULL DEFAULT '0,0,0,0,0' COMMENT 'time based sets comma separated' AFTER reps_sets, ADD IF NOT EXISTS time_based int NOT NULL DEFAULT '0' AFTER time_sets, CHANGE rest_period rest_period VARCHAR(25) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '30,30,30,30,30' COMMENT 'is rest between sets';" \
--assume-rbr \
--cut-over=default \
--exact-rowcount \
--concurrent-rowcount \
--initially-drop-ghost-table \
--default-retries=120 \
--postpone-cut-over-flag-file='/home/ubuntu/cut_over.flag' \
--hooks-path='/home/ubuntu/hooks' \
--conf='/home/ubuntu/gh-ost.cnf'
--allow-renamed-columns
```

content of gh-ost-on-before-cut-over-hook.sh:
```
#!/usr/bin/env bash

echo "Before cut-over started" >> /tmp/gh-ost.log

statement="UPDATE $GH_OST_GHOST_TABLE_NAME i SET i.reps_sets = CONCAT(i.set1,',',i.set2,',',i.set3,',',i.set4,',',i.set5),
rest_period = CONCAT(rest_period,',',rest_period,',',rest_period,',',rest_period,',',rest_period);"
echo "Issuing statement to update the ghost table before the cut-over: ${statement}" >> /tmp/gh-ost.log
exec mysql my_db_name -h production -uusername-ppassword -e "${statement}"
echo "Update finished" >> /tmp/gh-ost.log
```

So basically after the copy and before the switch, I wanted to do a big UPDATE.

I tried all first on a small table and both the copy, update and cutover (after deleting the flag file) went perfectly fine.

On the real table I issued a noop and everything went right, then a real --execute.
The process started correctly and it went on for 14h given the size of the table.

The problem is that this morning, for some reason, I woke up and gh-ost crashed.
I cannot find anymore the socket in /tmp or in any other place. The process just disappeared.

If I look in the db, I can see from the last line in the ghc table that the copy completed successfully:
```
240437 | 2017-08-04 01:17:05 | copy iteration 129266 at 1501809425 | Copy: 129265920/129265920 100.0%; Applied: 622862; Backlog: 0/1000; Time: 14h58m4s(total), 14h58m3s(copy); streamer: mysql-bin-changelog.000470:53163612; State: migrating; ETA: due
```
And my own /tmp/gh-ost.log doesn't contain the line:
`echo "Before cut-over started"`
so I can assume it never entered the before-cut-over phase.

Is there any way I can find more information about what happened?
Is there any way I can "resume" the process, instead of re-copy the table?

Thank you so much for the help and for this amazing tool!!

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia esaminando l'ultima voce della tabella ghc, i log disponibili in /tmp e l'output di gh-ost-on-before-cut-over-hook.sh in corrispondenza del completamento della copia segnalato. Traccia i punti di ingresso di cut-over e recovery per determinare perché il processo sia scomparso prima dell'hook e se la copia completata possa essere ripresa senza copiare di nuovo.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
go, mysql
Ambito
databases
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
32/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.