github / github/gh-ost

ERROR Timeout while waiting for events up to lock - after copy, well behind on binlog

Aperta
#709 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Go
Stelle
13.6k
Fork
1.4k
Merge medio
2h 31m
PR unite (30g)
4

Descrizione

I think this might be a little different than the other timeout, fail to cutover issues I'm seeing here.

I'm running gh-ost `1.0.47` on MySQL Aurora 1.16. Table I've been having problems with is about ~149G and about 5 million rows. Yeah, big table for so few rows. Had a gross text column.

This is the command I finally ran to successfully do the schema change:

```
gh-ost -allow-on-master -assume-rbr \
-critical-load Threads_running=600 -critical-load-hibernate-seconds 60 \
-database live_production -max-load Threads_running=400 -nice-ratio 0.1 \
-throttle-additional-flag-file ~/gh-ost-throttle_ghost \
--serve-socket-file=/tmp/gh-ost.sock \
-default-retries 1024 \
-chunk-size 5000 -ask-pass -user ${user} \
-table ${table} \
-host ${host} \
-alter 'ADD COLUMN `a` VARCHAR(3) DEFAULT NULL,
ADD COLUMN `b` INT DEFAULT NULL' \
-verbose -execute 2>&1 | tee gh-ost.log
```

Previous attempts were with less retries. After the copy was done, this was the binlog situation:

```
2019-02-12 04:13:17 INFO Waiting for events up to lock
Copy: 5242413/5242413 100.0%; Applied: 352; Backlog: 0/1000; Time: 2h40m10s(total), 2h35m47s(copy); streamer: mysql-bin-changelog.177290:164565360; State: migrating; ETA: due
2019-02-12 04:13:20 ERROR Timeout while waiting for events up to lock
2019-02-12 04:13:20 ERROR 2019-02-12 04:13:20 ERROR Timeout while waiting for events up to lock
```
```
mysql> show master status;
+----------------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+----------------------------+----------+--------------+------------------+-------------------+
| mysql-bin-changelog.177391 | 16275169 | | | |
+----------------------------+----------+--------------+------------------+-------------------+
```
So, as you can see, even though the backlog is 0, gh-ost is way behind on the binlog (177290 while 177391 was current). Almost 800 lock attempts latter, gh-ost finally caught up to current log and was able to successfully cut over.

Is this a bug or perhaps a parameters issue on my end? Thanks!

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia dal percorso del codice di gh-ost che emette “Waiting for events up to lock” e “Timeout while waiting for events up to lock”; confronta la sua posizione dell’evento con la copia fornita, il backlog e l’output di MySQL master-status. Usa il comando e i dettagli di Aurora dell’issue per determinare se il comportamento è causato dai parametri o da un difetto, quindi documenta la causa e la modifica necessaria.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
go, mysql
Ambito
databases
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.