StreamEvents encountered unexpected error ... Err: invalid json type 122
- Vorherrschende Sprache
- Go
- Sterne
- 13.6k
- Forks
- 1.4k
- Ø Merge
- 2 Std. 31 Min.
- Gemergte PRs (30 T.)
- 4
Beschreibung
We recently added some json datatype fields to our database . While running even a basic gh-ost migration (against a small , totally unrelated table of three records), we've run into issues with what appears to be gh-ost not being able to parse the json from the binlogs. The database values appear to be fine, other tools don't have an issue parsing or displaying them, and our production systems have no issue with the json we are creating), but as soon as there's a binlog event for one of these columns, gh-ost starts emitting errors and warnings and the migrations eventually stop processing.
We'll see the migration progressing as expected:
`
Copy: 10348000/15658439 66.1%; Applied: 0; Backlog: 0/1000; Time: 25m0s(total), 25m0s(copy); streamer: mysql-bin-changelog.059183:109285857; Lag: 0.11s, State: migrating; ETA: 12m49s
Copy: 10543000/15658439 67.3%; Applied: 0; Backlog: 0/1000; Time: 25m30s(total), 25m30s(copy); streamer: mysql-bin-changelog.059183:125594568; Lag: 0.11s, State: migrating; ETA: 12m22s
[2021/05/26 16:06:25] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.059184, 4)
[2021/05/26 16:06:25] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.059184, 4)
`
Then based on the output we are seeing (and we've replicated this in one of our labs) it looks like changes to the JSON values leads to errors like this:
`
2021-05-26 16:06:25 INFO rotate to next log from mysql-bin-changelog.059184:134246668 to mysql-bin-changelog.059184
2021-05-26 16:06:25 INFO rotate to next log from mysql-bin-changelog.059184:0 to mysql-bin-changelog.059184
Copy: 10733000/15658439 68.5%; Applied: 0; Backlog: 0/1000; Time: 26m0s(total), 26m0s(copy); streamer: mysql-bin-changelog.059184:7101260; Lag: 0.01s, State: migrating; ETA: 11m55s
[2021/05/26 16:07:03] [error] binlogstreamer.go:77 close sync with err: Header &replication.EventHeader{Timestamp:0x60aed497, EventType:0x1f, ServerID:0x6813be2c, EventSize:0x9ba, LogPos:0x132492d, Flags:0x0}, Data
, Err: invalid json type 122
`
followed by errors like:
`
2021-05-26 16:07:03 INFO StreamEvents encountered unexpected error: Header &replication.EventHeader{Timestamp:0x60aed497, EventType:0x1f, ServerID:0x6813be2c, EventSize:0x9ba, LogPos:0x132492d, Flags:0x0}, Data
", Err: invalid json type 122
`
and finally
`
FATAL 121 successive failures in streamer reconnect at coordinates mysql-bin-changelog.059184:4
`
We upgraded to gh-ost v1.1.1 and are running on Amazon Aurora MySQL version `5.7.mysql_aurora.2.08.0`
Has anyone seen this behaviour before?
Thank you kindly,
?d
Beitragsleitfaden
Rechercherichtung
Beginne bei binlogstreamer.go:77 und beim Rotationspfad in binlogsyncer.go um Zeile 723 und verfolge anschließend, wie der Fehler wegen des ungültigen JSON-Typs 122 behandelt wird. Reproduziere das gemeldete Ereignis mit dem zugesagten Labortestfall und bestätige, dass JSON-Binlog-Ereignisse während einer Migration nun weder das Streaming stoppen noch wiederholte Fehler beim erneuten Verbinden verursachen.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- go, mysql
- Bereich
- databases
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Veraltet
- Klarheit
- Muss geklärt werden
- Anfängerfreundlichkeit
- 25/100