airbytehq / airbytehq/airbyte

[source-mysql] CDC schema history grows unbounded due to missing `store.only.captured.tables.ddl` filtering

Đang mở
#74,002 3 bình luận 0 reaction 0 người được giao Xem trên GitHub
area/platform autoteam community connectors/source/mysql needs-triage team/compose team/platform-move type/bug
Ngôn ngữ chính
Python
Star
22.1k
Fork
5.3k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

### Helm Chart Version

2.0.19

### What step the error happened?

During the Sync

### Relevant information

### Connector Name

source-mysql

### Connector Version

3.51.5 (Debezium 3.1.2.Final)

### What step the error happened?

During incremental CDC syncs — progressive performance degradation over days/weeks.

### Problem

The MySQL CDC connector sets `schema.history.internal.store.only.captured.databases.ddl=true` to filter DDL by database, but does **not** set `schema.history.internal.store.only.captured.tables.ddl=true` to filter DDL by table.

This means the Debezium schema history accumulates DDL events for **every table in the database**, including tables that are not part of any configured stream. On databases with frequent DDL activity (e.g., application-generated temporary tables, sequence tables with `CREATE TABLE IF NOT EXISTS`), the schema history grows unboundedly and causes:

1. **Progressively slower sync startups** — Debezium replays the entire schema history at each sync start
2. **OOM crashes** — decompressed schema history exceeds available heap memory
3. **Silent failures** — syncs hit the heartbeat timeout during schema replay, are marked "succeeded" with 0 records

### Real-world impact

In our production environment, we replicate several MySQL 8 databases via CDC. Some databases have application patterns that generate high DDL volume on non-captured tables:

- **Pattern A**: A table used as a sequence generator is re-created with `CREATE TABLE IF NOT EXISTS` on every transaction — **~9,200 DDL/day**, all no-ops since the table already exists
- **Pattern B**: Temporary working tables are created and dropped per business unit per day — **~5,000 DDL/day**

None of these tables are configured as streams. Yet every DDL is stored in the schema history.

**Measured impact on hourly connections after just 17 days:**

| Connection schedule | Schema history entries | Compressed state size |
|---------------------|----------------------|----------------------|
| Hourly | **162,000+** | **10 MB** |
| Weekly (auto-recovery via binlog expiry) | **~640** | **254 KB** |

The weekly connection is naturally immune because the binlog position expires between syncs, forcing Debezium into `recovery` mode which rebuilds the schema from scratch. Hourly connections never enter recovery mode, so the history accumulates indefinitely.

After 17 days, `snapshot.mode=when_needed` replays 162K DDL entries at every sync startup (~70 seconds), vs ~2 seconds with a clean history.

### Root cause in code

**`DebeziumPropertiesBuilder.kt`** ([link](https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/bulk/toolkits/extract-cdc/src/main/kotlin/io/airbyte/cdk/read/cdc/DebeziumPropertiesBuilder.kt)):

```kotlin
fun withSchemaHistory(): DebeziumPropertiesBuilder = apply {
with("schema.history.internal", FileSchemaHistory::class.java.name)
with("schema.history.internal.store.only.captured.databases.ddl", "true")
// Missing: with("schema.history.internal.store.only.captured.tables.ddl", "true")
}
```

The property `schema.history.internal.store.only.captured.tables.ddl` is a [documented Debezium property](https://debezium.io/documentation/reference/stable/connectors/mysql.html) that restricts DDL storage to only tables in `table.include.list`. It exists precisely for this use case.

### Proposed fix

Add one line in `DebeziumPropertiesBuilder.withSchemaHistory()`:

```kotlin
fun withSchemaHistory(): DebeziumPropertiesBuilder = apply {
with("schema.history.internal", FileSchemaHistory::class.java.name)
with("schema.history.internal.store.only.captured.databases.ddl", "true")
with("schema.history.internal.store.only.captured.tables.ddl", "true") // ADD THIS
}
```

### Known tradeoff

When `store.only.captured.tables.ddl=true`, adding a new table to an existing connection requires a CDC state reset (because the new table's schema won't be in the history). This is already the case today in practice — see issue #28150 where adding tables fails with "schema isn't known to this connector". So this change would not introduce a new limitation, and the existing behavior of resetting state when adding tables would still work.

### Workaround

Currently the only workaround is to periodically reset the CDC state of affected connections via the API (`POST /api/v1/connections/reset`), which triggers a full re-sync of all data.

### Additional context

- Debezium community thread (2017) documenting the same issue: [very large ddl history topic](https://groups.google.com/g/debezium/c/QBxa5FstXS0)
- Confluent forum confirming the fix: [Filtering Specific Tables and Preventing Schema History](https://forum.confluent.io/t/debezium-connector-filtering-specific-tables-and-preventing-schema-history-for-all-tables/37456)
- This affects all database connectors using Debezium (MySQL, MariaDB, MSSQL), not just MySQL

### Relevant log output

```shell

```

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11421

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.