debezium / debezium/dbz

[MySQL] Introduce a lean table-schema-snapshot signal to dynamically capture schema for new tables

Open
#2,424 1 comment 0 reactions 1 assignee Claimed by @ShadowFox144 View on GitHub
component/core-library component/mysql-connector type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

### Describe the feature you'd like to request

Currently, when dynamically adding a new table to the MySQL connector's include list via a `PUT` API call with `schema.history.internal.store.only.captured.tables.ddl = true`, a "missing schema exception" is thrown.

To resolve this, the connector must run in recovery mode, which requires manually deleting the existing schema history topic so it can be recreated. Alternatively, setting the property to `false` triggers a long-running query on the database (which negatively impacts our AWS RDS Writer node in production).

While the existing `schema-changes` signal can act as a workaround, it is overly verbose as it requires supplying the `TableChanges` JSON in the exact format stored in the history topic.

**Proposed Solution:**
As discussed with @Naros and @vjuranek on Zulip, we should introduce a new, leaner signal (e.g., `table-schema-snapshot`).

This signal would only require a fully qualified table name in the payload. The connector would intercept this signal, pass the table identifier through to read the schema (similar to recovery mode), but **only for that specific table**.

**Ideal Workflow:**
1. Send the `table-schema-snapshot` signal with the FQDN of the new table.
2. Wait for the signal to be consumed and the schema to be captured.
3. Modify the connector's include list via API to start streaming.

### Is your feature request related to a problem? Please describe.

Yes, this solves a major operational bottleneck for dynamically scaling and onboarding new tables in MySQL CDC, specifically for environments where full schema dumps cause production database degradation.

### Describe alternatives considered

1. **Manual Topic Deletion:** Manually deleting the schema history topic and restarting the connector in recovery mode. (Not scalable).
2. **`schema-changes` signal:** Constructing the full `TableChanges` JSON manually. (Highly verbose and error-prone for users).

### Additional context

* **Zulip Discussion:** [#community-mysql-mariadb > Adding new tables to CDC monitoring without full schema](https://debezium.zulipchat.com/#narrow/channel/348104-community-mysql-mariadb/topic/Adding.20new.20tables.20to.20CDC.20monitoring.20without.20full.20schema/with/616453099)
* I am very interested in contributing this feature myself! Any pointers on where to start (e.g., existing signal implementation classes) would be highly appreciated.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.