github / github/gh-ost

StreamEvents encountered unexpected error ... Err: invalid json type 122

Aperta
#976 7 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Go
Stelle
13.6k
Fork
1.4k
Merge medio
2h 31m
PR unite (30g)
4

Descrizione

We recently added some json datatype fields to our database . While running even a basic gh-ost migration (against a small , totally unrelated table of three records), we've run into issues with what appears to be gh-ost not being able to parse the json from the binlogs. The database values appear to be fine, other tools don't have an issue parsing or displaying them, and our production systems have no issue with the json we are creating), but as soon as there's a binlog event for one of these columns, gh-ost starts emitting errors and warnings and the migrations eventually stop processing.

We'll see the migration progressing as expected:
`
Copy: 10348000/15658439 66.1%; Applied: 0; Backlog: 0/1000; Time: 25m0s(total), 25m0s(copy); streamer: mysql-bin-changelog.059183:109285857; Lag: 0.11s, State: migrating; ETA: 12m49s
Copy: 10543000/15658439 67.3%; Applied: 0; Backlog: 0/1000; Time: 25m30s(total), 25m30s(copy); streamer: mysql-bin-changelog.059183:125594568; Lag: 0.11s, State: migrating; ETA: 12m22s
[2021/05/26 16:06:25] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.059184, 4)
[2021/05/26 16:06:25] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.059184, 4)
`

Then based on the output we are seeing (and we've replicated this in one of our labs) it looks like changes to the JSON values leads to errors like this:
`
2021-05-26 16:06:25 INFO rotate to next log from mysql-bin-changelog.059184:134246668 to mysql-bin-changelog.059184
2021-05-26 16:06:25 INFO rotate to next log from mysql-bin-changelog.059184:0 to mysql-bin-changelog.059184
Copy: 10733000/15658439 68.5%; Applied: 0; Backlog: 0/1000; Time: 26m0s(total), 26m0s(copy); streamer: mysql-bin-changelog.059184:7101260; Lag: 0.01s, State: migrating; ETA: 11m55s
[2021/05/26 16:07:03] [error] binlogstreamer.go:77 close sync with err: Header &replication.EventHeader{Timestamp:0x60aed497, EventType:0x1f, ServerID:0x6813be2c, EventSize:0x9ba, LogPos:0x132492d, Flags:0x0}, Data

, Err: invalid json type 122
`

followed by errors like:
`
2021-05-26 16:07:03 INFO StreamEvents encountered unexpected error: Header &replication.EventHeader{Timestamp:0x60aed497, EventType:0x1f, ServerID:0x6813be2c, EventSize:0x9ba, LogPos:0x132492d, Flags:0x0}, Data

", Err: invalid json type 122
`

and finally
`
FATAL 121 successive failures in streamer reconnect at coordinates mysql-bin-changelog.059184:4
`

We upgraded to gh-ost v1.1.1 and are running on Amazon Aurora MySQL version `5.7.mysql_aurora.2.08.0`

Has anyone seen this behaviour before?

Thank you kindly,

?d

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia da binlogstreamer.go:77 e dal percorso di rotazione di binlogsyncer.go intorno alla riga 723, quindi traccia la gestione dell'errore relativo al tipo JSON 122 non valido. Riproduci l'evento segnalato con il caso di test di laboratorio promesso e conferma che gli eventi binlog JSON non interrompono più lo streaming né causano errori ripetuti di riconnessione durante una migrazione.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
go, mysql
Ambito
databases
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.