[source-mysql] CDC schema history grows unbounded due to missing `store.only.captured.tables.ddl` filtering
- Ngôn ngữ chính
- Python
- Star
- 22.1k
- Fork
- 5.3k
- Chỉ số merge pull request
- Chỉ số pull request đang chờ
Mô tả
### Helm Chart Version
2.0.19
### What step the error happened?
During the Sync
### Relevant information
### Connector Name
source-mysql
### Connector Version
3.51.5 (Debezium 3.1.2.Final)
### What step the error happened?
During incremental CDC syncs — progressive performance degradation over days/weeks.
### Problem
The MySQL CDC connector sets `schema.history.internal.store.only.captured.databases.ddl=true` to filter DDL by database, but does **not** set `schema.history.internal.store.only.captured.tables.ddl=true` to filter DDL by table.
This means the Debezium schema history accumulates DDL events for **every table in the database**, including tables that are not part of any configured stream. On databases with frequent DDL activity (e.g., application-generated temporary tables, sequence tables with `CREATE TABLE IF NOT EXISTS`), the schema history grows unboundedly and causes:
1. **Progressively slower sync startups** — Debezium replays the entire schema history at each sync start
2. **OOM crashes** — decompressed schema history exceeds available heap memory
3. **Silent failures** — syncs hit the heartbeat timeout during schema replay, are marked "succeeded" with 0 records
### Real-world impact
In our production environment, we replicate several MySQL 8 databases via CDC. Some databases have application patterns that generate high DDL volume on non-captured tables:
- **Pattern A**: A table used as a sequence generator is re-created with `CREATE TABLE IF NOT EXISTS` on every transaction — **~9,200 DDL/day**, all no-ops since the table already exists
- **Pattern B**: Temporary working tables are created and dropped per business unit per day — **~5,000 DDL/day**
None of these tables are configured as streams. Yet every DDL is stored in the schema history.
**Measured impact on hourly connections after just 17 days:**
| Connection schedule | Schema history entries | Compressed state size |
|---------------------|----------------------|----------------------|
| Hourly | **162,000+** | **10 MB** |
| Weekly (auto-recovery via binlog expiry) | **~640** | **254 KB** |
The weekly connection is naturally immune because the binlog position expires between syncs, forcing Debezium into `recovery` mode which rebuilds the schema from scratch. Hourly connections never enter recovery mode, so the history accumulates indefinitely.
After 17 days, `snapshot.mode=when_needed` replays 162K DDL entries at every sync startup (~70 seconds), vs ~2 seconds with a clean history.
### Root cause in code
**`DebeziumPropertiesBuilder.kt`** ([link](https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/bulk/toolkits/extract-cdc/src/main/kotlin/io/airbyte/cdk/read/cdc/DebeziumPropertiesBuilder.kt)):
```kotlin
fun withSchemaHistory(): DebeziumPropertiesBuilder = apply {
with("schema.history.internal", FileSchemaHistory::class.java.name)
with("schema.history.internal.store.only.captured.databases.ddl", "true")
// Missing: with("schema.history.internal.store.only.captured.tables.ddl", "true")
}
```
The property `schema.history.internal.store.only.captured.tables.ddl` is a [documented Debezium property](https://debezium.io/documentation/reference/stable/connectors/mysql.html) that restricts DDL storage to only tables in `table.include.list`. It exists precisely for this use case.
### Proposed fix
Add one line in `DebeziumPropertiesBuilder.withSchemaHistory()`:
```kotlin
fun withSchemaHistory(): DebeziumPropertiesBuilder = apply {
with("schema.history.internal", FileSchemaHistory::class.java.name)
with("schema.history.internal.store.only.captured.databases.ddl", "true")
with("schema.history.internal.store.only.captured.tables.ddl", "true") // ADD THIS
}
```
### Known tradeoff
When `store.only.captured.tables.ddl=true`, adding a new table to an existing connection requires a CDC state reset (because the new table's schema won't be in the history). This is already the case today in practice — see issue #28150 where adding tables fails with "schema isn't known to this connector". So this change would not introduce a new limitation, and the existing behavior of resetting state when adding tables would still work.
### Workaround
Currently the only workaround is to periodically reset the CDC state of affected connections via the API (`POST /api/v1/connections/reset`), which triggers a full re-sync of all data.
### Additional context
- Debezium community thread (2017) documenting the same issue: [very large ddl history topic](https://groups.google.com/g/debezium/c/QBxa5FstXS0)
- Confluent forum confirming the fix: [Filtering Specific Tables and Preventing Schema History](https://forum.confluent.io/t/debezium-connector-filtering-specific-tables-and-preventing-schema-history-for-all-tables/37456)
- This affects all database connectors using Debezium (MySQL, MariaDB, MSSQL), not just MySQL
### Relevant log output
```shell
```
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11421
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.