debezium / debezium/dbz

MySql schema change support [DBZ-2735]

Open
#384 0 comments 0 reactions 0 assignees View on GitHub
component/mysql-connector migrated-from-jira type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-2735](https://issues.redhat.com/browse/DBZ-2735)

Hello! 

we have been using the MySQL Debezium connector in order to synchronize our production MySQL table with our Snowflake data warehouse.

Some of our production tables store hundreds of millions of records, whenever we need to alter the schema of any of those tables we use a Percona script [pt-online-schema-change|[https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html#description]. The script emulates the way that MySQL alters tables internally, but it works on a copy of the table you wish to alter. This means that the original table is not locked, and clients may continue to read and change data in it.

From Debezium's perspective, the following process will create a queue of changes in the MySQL binlog that will cause our incoming CDC data from Debezium to be delayed by up to days or weeks. The connector is treating the alter table process as new records inserted into the table as the copy of records from the old altered table to the new schema table as new records that need to be streamed to Snowflake.

We are looking for a way to ignore those records, without a manual intervention where we will need to go and change the connectors offset. If there was a way to signal the connector to ignore records for a certain table until we are done with the schema table changes that would be great.

 

Let us know if you have any questions

FYI [~stephen_marin] [~gunnar.morling] [~jpechane] 

 

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by examining the MySQL connector's binlog handling for pt-online-schema-change events and how table schema changes are represented. Done should mean records copied during the schema change can be ignored for a selected table without manually changing connector offsets, while normal changes remain streamed.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.