debezium / debezium/dbz

Incremental snapshot - wrong table name in offsets [DBZ-8567]

Open
#61 0 comments 0 reactions 1 assignee Claimed by @mfvitale View on GitHub
component/sqlserver-connector migrated-from-jira type/bug
Dominant language
HTML
Stars
6
Forks
9
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-8567](https://issues.redhat.com/browse/DBZ-8567)

h1. Bug report
h2. What Debezium connector do you use and what version?

DBZ SQL Server Connector 2.5.4
h2. What is the connector configuration?

{     "name": "",     "connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",     "tasks.max": "1",     "topic.prefix": "",     "key.converter": "org.apache.kafka.connect.json.JsonConverter",     "value.converter": "org.apache.kafka.connect.json.JsonConverter",     "database.encrypt": "false",     "database.user": "",     "database.names": "",     "database.hostname": "",     "database.password": "",     "table.include.list": "dbo.Process,dbo.Sensor",     "snapshot.mode": "schema_only",     "snapshot.isolation.mode": "snapshot",     "incremental.snapshot.chunk.size": "50000",     "signal.enabled.channels": "source",     "signal.data.collection": "",     "signal.kafka.bootstrap.servers": "",     "binary.handling.mode": "hex",     "transforms": "addTopicSuffix,unwrap",     "transforms.addTopicSuffix.type": "org.apache.kafka.connect.transforms.RegexRouter",     "transforms.addTopicSuffix.replacement": "$0.raw",     "transforms.addTopicSuffix.regex": ".*",     "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",     "transforms.unwrap.delete.tombstone.handling.mode": "rewrite",     "schema.history.internal.kafka.topic": "",     "schema.history.internal.kafka.bootstrap.servers": "",     "schema.history.internal.producer.sasl.mechanism": "PLAIN",     "schema.history.internal.producer.security.protocol": "SASL_SSL",     "schema.history.internal.producer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\" password=\"\";",     "schema.history.internal.consumer.sasl.mechanism": "PLAIN",     "schema.history.internal.consumer.security.protocol": "SASL_SSL",     "schema.history.internal.consumer.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\" password=\"\";"      }
h2. What is the captured database version and mode of deployment?

on-premise, MS SQL Server 2017/2019
h2. What behavior do you expect?

When initiating an incremental snapshot (by inserting a signal in the database signal table) the connector should snapshot the specified table and write the offsets correctly in the Kafka Connect offset topic. Upon inserting another signal for a different table, the second table should be snapshotted, and its offsets should also be written correctly to the Kafka Connect offset topic.

h2. What behavior do you see?

When snapshotting multiple tables consecutively, the Kafka Connect offset topic records the correct (max/last) primary key but retains the table name of the first table, even when a different table is snapshotted.

This is usually not a problem, but when the connector is reset, couple of problems can happen, since it tries to recover by reading the last message from the kafka connect offset topic - since topic has the correct primary key of the table it was reading, but the wrong table name:
# error related to the primary key mismatch - usually due to composite key being partially or incorrectly applied - examples of both are in "Snapshot errors" attachment:
## index out of bounds exception
## value not set for the parameter x
# If the primary keys of the tables are similar in structure, the connector might begin reading the first table from the point where the second table left off, instead of continue reading the second table.

 
h2. Do you see the same behaviour using the latest released Debezium version?

havent tried
h2. Do you have the connector logs, ideally from start till finish?

(You might be asked later to provide [DEBUG/TRACE|https://debezium.io/documentation/reference/stable/operations/logging.html] level log)

Added examples (attached in image and txt format) of different exceptions occuring after connector has been restarted during incremental snapshot of a (second) table.
h2. How to reproduce the issue using our [tutorial|https://github.com/debezium/debezium-examples/tree/main/tutorial] deployment?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.