github / github/gh-ost

StreamEvents encountered unexpected error ... Err: invalid json type 122

Offen
#976 7 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Go
Sterne
13.6k
Forks
1.4k
Ø Merge
2 Std. 31 Min.
Gemergte PRs (30 T.)
4

Beschreibung

We recently added some json datatype fields to our database . While running even a basic gh-ost migration (against a small , totally unrelated table of three records), we've run into issues with what appears to be gh-ost not being able to parse the json from the binlogs. The database values appear to be fine, other tools don't have an issue parsing or displaying them, and our production systems have no issue with the json we are creating), but as soon as there's a binlog event for one of these columns, gh-ost starts emitting errors and warnings and the migrations eventually stop processing.

We'll see the migration progressing as expected:
`
Copy: 10348000/15658439 66.1%; Applied: 0; Backlog: 0/1000; Time: 25m0s(total), 25m0s(copy); streamer: mysql-bin-changelog.059183:109285857; Lag: 0.11s, State: migrating; ETA: 12m49s
Copy: 10543000/15658439 67.3%; Applied: 0; Backlog: 0/1000; Time: 25m30s(total), 25m30s(copy); streamer: mysql-bin-changelog.059183:125594568; Lag: 0.11s, State: migrating; ETA: 12m22s
[2021/05/26 16:06:25] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.059184, 4)
[2021/05/26 16:06:25] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.059184, 4)
`

Then based on the output we are seeing (and we've replicated this in one of our labs) it looks like changes to the JSON values leads to errors like this:
`
2021-05-26 16:06:25 INFO rotate to next log from mysql-bin-changelog.059184:134246668 to mysql-bin-changelog.059184
2021-05-26 16:06:25 INFO rotate to next log from mysql-bin-changelog.059184:0 to mysql-bin-changelog.059184
Copy: 10733000/15658439 68.5%; Applied: 0; Backlog: 0/1000; Time: 26m0s(total), 26m0s(copy); streamer: mysql-bin-changelog.059184:7101260; Lag: 0.01s, State: migrating; ETA: 11m55s
[2021/05/26 16:07:03] [error] binlogstreamer.go:77 close sync with err: Header &replication.EventHeader{Timestamp:0x60aed497, EventType:0x1f, ServerID:0x6813be2c, EventSize:0x9ba, LogPos:0x132492d, Flags:0x0}, Data

, Err: invalid json type 122
`

followed by errors like:
`
2021-05-26 16:07:03 INFO StreamEvents encountered unexpected error: Header &replication.EventHeader{Timestamp:0x60aed497, EventType:0x1f, ServerID:0x6813be2c, EventSize:0x9ba, LogPos:0x132492d, Flags:0x0}, Data

", Err: invalid json type 122
`

and finally
`
FATAL 121 successive failures in streamer reconnect at coordinates mysql-bin-changelog.059184:4
`

We upgraded to gh-ost v1.1.1 and are running on Amazon Aurora MySQL version `5.7.mysql_aurora.2.08.0`

Has anyone seen this behaviour before?

Thank you kindly,

?d

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne bei binlogstreamer.go:77 und beim Rotationspfad in binlogsyncer.go um Zeile 723 und verfolge anschließend, wie der Fehler wegen des ungültigen JSON-Typs 122 behandelt wird. Reproduziere das gemeldete Ereignis mit dem zugesagten Labortestfall und bestätige, dass JSON-Binlog-Ereignisse während einer Migration nun weder das Streaming stoppen noch wiederholte Fehler beim erneuten Verbinden verursachen.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
go, mysql
Bereich
databases
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.