apache / apache/doris-flink-connector
[Enhancement] Improve automatic DDL handling for full-database sync in CdcTools
- Dominant language
- Java
- Stars
- 385
- Forks
- 279
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 6
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
### 📝 **Description**
When using `CdcTools` with the `oracle-sync-database` mode for full-database synchronization,
the automatic DDL handling is still not stable under the `--schema-change-mode debezium_structure` mode.
We are using this mode intentionally to support multiple heterogeneous data sources (Oracle, MySQL, PostgreSQL, etc.), since `sql_parser` mode is not fully compatible with all of them.
However, when schema changes (DDL) occur — especially frequent `ALTER TABLE ADD/DROP COLUMN` — Doris sometimes fails to synchronize the schema correctly.
### ⚙️ **Current Behavior**
* Some DDL operations (such as `ADD COLUMN` or `DROP COLUMN`) are not correctly applied to Doris tables.
* When multiple DDLs occur within a short period, Doris schema can go out of sync with the source.
* The connector logs occasionally show parsing or schema errors like:
> `"The recordRoot cannot extract ddl sql"`
> `"Column does not exist"` when Doris attempts to drop an already-dropped column.
* These issues seem more frequent under `debezium_structure` mode.
### ✅ **Expected Behavior**
* DDL synchronization should be consistent and idempotent even when multiple DDLs occur rapidly.
* Doris should gracefully skip redundant DDLs (for example, when a column already exists or is already dropped).
* The `debezium_structure` mode should provide better schema evolution handling across multiple database dialects.
* Ideally, `CdcTools` could include retry or verification mechanisms to validate Doris schema alignment after each DDL event.
### 🧩 **Environment**
* Doris Flink Connector: 1.18-25.1.0
* Flink Version: 1.18.0
* Oracle Source: 19c (LogMiner)
* Schema Change Mode: `debezium_structure`
* Sync Mode: `oracle-sync-database` (CdcTools)
* Sink: Doris 2.x
*
### 🧾 **Example Log**
schema change status:true, ddl: ALTER TABLE `test_db`.`EX_CTRL_CARE_PLAN` DROP COLUMN `PLAN_TYPE`
schema change error : Failed to execute sql: Column does not exist: PLAN_TYPE
The recordRoot cannot extract ddl sql
### Solution
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start by tracing the CdcTools oracle-sync-database path and its debezium_structure schema-change handling, using the reported ALTER TABLE logs as the reproduction signal. Compare rapid ADD/DROP COLUMN events and define completion as consistent, idempotent Doris schema synchronization without redundant-column failures or recordRoot DDL extraction errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100