apache / apache/doris-flink-connector

[Enhancement] Improve automatic DDL handling for full-database sync in CdcTools

Open
#618 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
385
Forks
279
Avg merge
2d 2h
Merged PRs (30d)
6

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Description

### 📝 **Description**
When using `CdcTools` with the `oracle-sync-database` mode for full-database synchronization,
the automatic DDL handling is still not stable under the `--schema-change-mode debezium_structure` mode.
We are using this mode intentionally to support multiple heterogeneous data sources (Oracle, MySQL, PostgreSQL, etc.), since `sql_parser` mode is not fully compatible with all of them.
However, when schema changes (DDL) occur — especially frequent `ALTER TABLE ADD/DROP COLUMN` — Doris sometimes fails to synchronize the schema correctly.

### ⚙️ **Current Behavior**
* Some DDL operations (such as `ADD COLUMN` or `DROP COLUMN`) are not correctly applied to Doris tables.
* When multiple DDLs occur within a short period, Doris schema can go out of sync with the source.
* The connector logs occasionally show parsing or schema errors like:
> `"The recordRoot cannot extract ddl sql"`
> `"Column does not exist"` when Doris attempts to drop an already-dropped column.
* These issues seem more frequent under `debezium_structure` mode.

### ✅ **Expected Behavior**
* DDL synchronization should be consistent and idempotent even when multiple DDLs occur rapidly.
* Doris should gracefully skip redundant DDLs (for example, when a column already exists or is already dropped).
* The `debezium_structure` mode should provide better schema evolution handling across multiple database dialects.
* Ideally, `CdcTools` could include retry or verification mechanisms to validate Doris schema alignment after each DDL event.

### 🧩 **Environment**
* Doris Flink Connector: 1.18-25.1.0
* Flink Version: 1.18.0
* Oracle Source: 19c (LogMiner)
* Schema Change Mode: `debezium_structure`
* Sync Mode: `oracle-sync-database` (CdcTools)
* Sink: Doris 2.x
*
### 🧾 **Example Log**
schema change status:true, ddl: ALTER TABLE `test_db`.`EX_CTRL_CARE_PLAN` DROP COLUMN `PLAN_TYPE`
schema change error : Failed to execute sql: Column does not exist: PLAN_TYPE
The recordRoot cannot extract ddl sql

### Solution

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the CdcTools oracle-sync-database path and its debezium_structure schema-change handling, using the reported ALTER TABLE logs as the reproduction signal. Compare rapid ADD/DROP COLUMN events and define completion as consistent, idempotent Doris schema synchronization without redundant-column failures or recordRoot DDL extraction errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.