[MySQL] Introduce a lean table-schema-snapshot signal to dynamically capture schema for new tables
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
### Describe the feature you'd like to request
Currently, when dynamically adding a new table to the MySQL connector's include list via a `PUT` API call with `schema.history.internal.store.only.captured.tables.ddl = true`, a "missing schema exception" is thrown.
To resolve this, the connector must run in recovery mode, which requires manually deleting the existing schema history topic so it can be recreated. Alternatively, setting the property to `false` triggers a long-running query on the database (which negatively impacts our AWS RDS Writer node in production).
While the existing `schema-changes` signal can act as a workaround, it is overly verbose as it requires supplying the `TableChanges` JSON in the exact format stored in the history topic.
**Proposed Solution:**
As discussed with @Naros and @vjuranek on Zulip, we should introduce a new, leaner signal (e.g., `table-schema-snapshot`).
This signal would only require a fully qualified table name in the payload. The connector would intercept this signal, pass the table identifier through to read the schema (similar to recovery mode), but **only for that specific table**.
**Ideal Workflow:**
1. Send the `table-schema-snapshot` signal with the FQDN of the new table.
2. Wait for the signal to be consumed and the schema to be captured.
3. Modify the connector's include list via API to start streaming.
### Is your feature request related to a problem? Please describe.
Yes, this solves a major operational bottleneck for dynamically scaling and onboarding new tables in MySQL CDC, specifically for environments where full schema dumps cause production database degradation.
### Describe alternatives considered
1. **Manual Topic Deletion:** Manually deleting the schema history topic and restarting the connector in recovery mode. (Not scalable).
2. **`schema-changes` signal:** Constructing the full `TableChanges` JSON manually. (Highly verbose and error-prone for users).
### Additional context
* **Zulip Discussion:** [#community-mysql-mariadb > Adding new tables to CDC monitoring without full schema](https://debezium.zulipchat.com/#narrow/channel/348104-community-mysql-mariadb/topic/Adding.20new.20tables.20to.20CDC.20monitoring.20without.20full.20schema/with/616453099)
* I am very interested in contributing this feature myself! Any pointers on where to start (e.g., existing signal implementation classes) would be highly appreciated.
Contributor guide
Assessment
This issue has not been assessed yet.