github / github/gh-ost

StreamEvents encountered unexpected error ... Err: invalid json type 122

Open
#976 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
13.6k
Forks
1.4k
Avg merge
2h 31m
Merged PRs (30d)
4

Description

We recently added some json datatype fields to our database . While running even a basic gh-ost migration (against a small , totally unrelated table of three records), we've run into issues with what appears to be gh-ost not being able to parse the json from the binlogs. The database values appear to be fine, other tools don't have an issue parsing or displaying them, and our production systems have no issue with the json we are creating), but as soon as there's a binlog event for one of these columns, gh-ost starts emitting errors and warnings and the migrations eventually stop processing.

We'll see the migration progressing as expected:
`
Copy: 10348000/15658439 66.1%; Applied: 0; Backlog: 0/1000; Time: 25m0s(total), 25m0s(copy); streamer: mysql-bin-changelog.059183:109285857; Lag: 0.11s, State: migrating; ETA: 12m49s
Copy: 10543000/15658439 67.3%; Applied: 0; Backlog: 0/1000; Time: 25m30s(total), 25m30s(copy); streamer: mysql-bin-changelog.059183:125594568; Lag: 0.11s, State: migrating; ETA: 12m22s
[2021/05/26 16:06:25] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.059184, 4)
[2021/05/26 16:06:25] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.059184, 4)
`

Then based on the output we are seeing (and we've replicated this in one of our labs) it looks like changes to the JSON values leads to errors like this:
`
2021-05-26 16:06:25 INFO rotate to next log from mysql-bin-changelog.059184:134246668 to mysql-bin-changelog.059184
2021-05-26 16:06:25 INFO rotate to next log from mysql-bin-changelog.059184:0 to mysql-bin-changelog.059184
Copy: 10733000/15658439 68.5%; Applied: 0; Backlog: 0/1000; Time: 26m0s(total), 26m0s(copy); streamer: mysql-bin-changelog.059184:7101260; Lag: 0.01s, State: migrating; ETA: 11m55s
[2021/05/26 16:07:03] [error] binlogstreamer.go:77 close sync with err: Header &replication.EventHeader{Timestamp:0x60aed497, EventType:0x1f, ServerID:0x6813be2c, EventSize:0x9ba, LogPos:0x132492d, Flags:0x0}, Data

, Err: invalid json type 122
`

followed by errors like:
`
2021-05-26 16:07:03 INFO StreamEvents encountered unexpected error: Header &replication.EventHeader{Timestamp:0x60aed497, EventType:0x1f, ServerID:0x6813be2c, EventSize:0x9ba, LogPos:0x132492d, Flags:0x0}, Data

", Err: invalid json type 122
`

and finally
`
FATAL 121 successive failures in streamer reconnect at coordinates mysql-bin-changelog.059184:4
`

We upgraded to gh-ost v1.1.1 and are running on Amazon Aurora MySQL version `5.7.mysql_aurora.2.08.0`

Has anyone seen this behaviour before?

Thank you kindly,

?d

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.