MySql schema change support [DBZ-2735]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-2735](https://issues.redhat.com/browse/DBZ-2735)
Hello!
we have been using the MySQL Debezium connector in order to synchronize our production MySQL table with our Snowflake data warehouse.
Some of our production tables store hundreds of millions of records, whenever we need to alter the schema of any of those tables we use a Percona script [pt-online-schema-change|[https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html#description]. The script emulates the way that MySQL alters tables internally, but it works on a copy of the table you wish to alter. This means that the original table is not locked, and clients may continue to read and change data in it.
From Debezium's perspective, the following process will create a queue of changes in the MySQL binlog that will cause our incoming CDC data from Debezium to be delayed by up to days or weeks. The connector is treating the alter table process as new records inserted into the table as the copy of records from the old altered table to the new schema table as new records that need to be streamed to Snowflake.
We are looking for a way to ignore those records, without a manual intervention where we will need to go and change the connectors offset. If there was a way to signal the connector to ignore records for a certain table until we are done with the schema table changes that would be great.
Let us know if you have any questions
FYI [~stephen_marin] [~gunnar.morling] [~jpechane]
Contributor guide
Research direction
No files or tests are named. Start by examining the MySQL connector's binlog handling for pt-online-schema-change events and how table schema changes are represented. Done should mean records copied during the schema change can be ignored for a selected table without manually changing connector offsets, while normal changes remain streamed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100