debezium / debezium/dbz

Internal schema may get corrupted due to duplicates in database history topic [DBZ-5342]

Open
#653 0 comments 0 reactions 0 assignees View on GitHub
component/core-library migrated-from-jira type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-5342](https://issues.redhat.com/browse/DBZ-5342)

In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
h1. Bug report

For bug reports, provide this information, please:
h2. What Debezium connector do you use and what version?

v1.9.2.Final
h2. What is the connector configuration?

This is a design issue unrelated to the configuration.
h2. What is the captured database version and mode of depoyment?

Irrelevant
h2. What behaviour do you expect?

The connector ignores duplicates in the database schema history.
h2. What behaviour do you see?

The connector consumes duplicates from the database schema history which causes schema corruption.
h2. Do you see the same behaviour using the latest relesead Debezium version?

I haven't tested it but the likely exists in all versions since it's by design.
h2. Do you have the connector logs, ideally from start till finish?

No.
h2. How to reproduce the issue using our [tutorial|https://github.com/debezium/debezium-examples/tree/main/tutorial] deployment?

Reproducing the issue requires an emergency shutdown of the Kafka Connect worker (e.g. due to an out-of-memory issue). It's hard to reproduce it in a reliable way. See details in the steps to reproduce.
h1. Feature request or enhancement
h2. Which use case/requirement will be addressed by the proposed feature?

Connector stability.
h2. Implementation ideas (optional)
# Make the keys of schema change history globally unique by adding the sequence number within the binlog event to the schema change metadata.
# When recovering schema from history, keep track of the binlog filename, position, and sequence number of the last processed schema change. Ignore all changes with the binlog filename, position, and sequence number less or equal to the last processed thereby deduplicating history.

{*}TODO{*}:

Is the source metadata of schema change events of other connectors sufficient for deduplication? If not, what do we do to address the issue in the MySQL connector without breaking others?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.