airbytehq / airbytehq/airbyte

Source MySQL: Encountered change event for table whose schema isn't known to this connector

Offen
#28,150 15 Kommentare 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen
area/connectors community connectors/source/mysql connectors/sources-database frozen releaseStage/beta team/db-dw-sources type/bug
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

### Connector Name

source-mysql

### Connector Version

2.0.19

### What step the error happened?

During the sync

### Revelant information

When we add a new table to the existing connection after a while it fails with Debezium exception
```
Encountered change event for table dynamodb-clone.BillingHourType whose schema isn't known to this connector
```

It seems like the connector does not propagate schema for the new table into the state: there is no ddl key for the new table.
The workaround is resetting the whole connection whenever you add a new table.

### Relevant log output

```
2023-07-10 19:10:13 [44msource[0m > ERROR i.d.c.m.MySqlStreamingChangeEventSource(informAboutUnknownTableIfRequired):692 Encountered change event 'Event{header=EventHeaderV4{timestamp=1689015633000, eventType=TABLE_MAP, serverId=780484762, headerLength=19, dataLength=555, nextPosition=32152271, flags=0}, data=TableMapEventData{tableId=278, database='dynamodb-clone', table='BillingHourType', columnTypes=-4, -4, 1, 15, -4, -4, 15, 15, 3, 15, 15, 15, -4, 15, 8, 8, 15, 8, 8, 15, 8, 8, 15, 15, 15, 15, 15, 15, 15, 15, 3, 3, 1, 8, 3, 18, 17, 1, columnMetadata=3, 3, 0, 144, 3, 3, 144, 400, 0, 144, 144, 144, 3, 1020, 0, 0, 1020, 0, 0, 1020, 0, 0, 1020, 1020, 1020, 1020, 1020, 1020, 1020, 1020, 0, 0, 0, 0, 0, 0, 6, 0, columnNullability={0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 37}, eventMetadata=TableMapEventMetadata{signedness={}, defaultCharset=255, charsetCollations=null, columnCharsets=null, columnNames=hourType, hourTypeAbbreviation, isActive, parentId, parentSortKey, hierarchy, id, entityType, version, tenantId, tenantCompanyId, partitionKey, sortKey, createdBy, createdDate, createdDateTime, deletedBy, deletedDate, deletedDateTime, lastUpdatedBy, lastUpdatedDate, lastUpdatedDateTime, lsi1, lsi2, lsi3, lsi4, lsi5, gsi1, gsi2, gsi3, rangeKey, _version, _deleted, _lastChangedAt, sortOrder, offlineUpdatedDateTime, lastModifiedUTC, isArchived, setStrValues=null, enumStrValues=null, geometryTypes=null, simplePrimaryKeys=6, primaryKeysWithPrefix=null, enumAndSetDefaultCharset=null, enumAndSetColumnCharsets=null,visibility=null}}}' at offset {transaction_id=null, ts_sec=1689015633, file=mysql-bin-changelog.009578, pos=32151597, server_id=780484762, event=1} for table dynamodb-clone.BillingHourType whose schema isn't known to this connector. One possible cause is an incomplete database schema history topic. Take a new snapshot in this case.
Use the mysqlbinlog tool to view the problematic event: mysqlbinlog --start-position=32151697 --stop-position=32152271 --verbose mysql-bin-changelog.009578
2023-07-10 19:10:13 [44msource[0m > ERROR i.d.c.m.MySqlStreamingChangeEventSource(logStreamingSourceState):1143 Error during binlog processing. Last offset stored = {transaction_id=null, ts_sec=1689015633, file=mysql-bin-changelog.009578, pos=32151597, server_id=780484762, event=1}, binlog reader near position = mysql-bin-changelog.009578/32151697
2023-07-10 19:10:13 [44msource[0m > ERROR i.d.p.ErrorHandler(setProducerThrowable):57 Producer failure io.debezium.DebeziumException: Error processing binlog event
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:392) ~[debezium-connector-mysql-2.2.0.Final.jar:2.2.0.Final]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$27(MySqlStreamingChangeEventSource.java:923) ~[debezium-connector-mysql-2.2.0.Final.jar:2.2.0.Final]
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1246) ~[mysql-binlog-connector-java-0.27.2.jar:0.27.2]
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1072) ~[mysql-binlog-connector-java-0.27.2.jar:0.27.2]
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631) ~[mysql-binlog-connector-java-0.27.2.jar:0.27.2]
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932) ~[mysql-binlog-connector-java-0.27.2.jar:0.27.2]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.debezium.DebeziumException: Encountered change event for table dynamodb-clone.BillingHourType whose schema isn't known to this connector
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:697) ~[debezium-connector-mysql-2.2.0.Final.jar:2.2.0.Final]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:738) ~[debezium-connector-mysql-2.2.0.Final.jar:2.2.0.Final]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:653) ~[debezium-connector-mysql-2.2.0.Final.jar:2.2.0.Final]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$14(MySqlStreamingChangeEventSource.java:893) ~[debezium-connector-mysql-2.2.0.Final.jar:2.2.0.Final]
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:372) ~[debezium-connector-mysql-2.2.0.Final.jar:2.2.0.Final]
... 6 more

Stack Trace: io.debezium.DebeziumException: Error processing binlog event
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:392)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$27(MySqlStreamingChangeEventSource.java:923)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1246)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1072)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.debezium.DebeziumException: Encountered change event for table dynamodb-clone.BillingHourType whose schema isn't known to this connector
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:697)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:738)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:653)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$14(MySqlStreamingChangeEventSource.java:893)
at io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:372)
... 6 more
```

### Contribute

- [ ] Yes, I want to contribute

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.