apache / apache/seatunnel

How to implement over‑write (clear the target table) during multi‑table synchronization? custom_sql cannot dynamically adapt to each table name.

Open
#11,802 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

环境信息
SeaTunnel 版本:2.3.18
Doris版本: 3.x
Doris Connector 版本:connector-doris-2.3.18.jar
部署方式:集群模式(Master/Worker)
JDK 版本:1.8

问题描述
在使用 Doris Sink 进行多表同步时,需要实现“覆盖写入”(即每次任务启动前先清空目标表,再写入数据),以保证同步的幂等性。
在单表同步时,我可以通过以下配置实现:

hocon
Doris {
...
data_save_mode = "CUSTOM_PROCESSING"
custom_sql = "TRUNCATE TABLE test.table1"
sink.enable-delete = "true"
}
该配置能在每个 Stream Load 批次前执行指定的 TRUNCATE 语句,清空目标表,效果正常。

但在多表同步场景中(例如使用 table_list 或通配符匹配多张表),custom_sql 是静态字符串,无法根据当前写入的表名动态生成对应的 TRUNCATE 语句。即使配置了 data_save_mode = "CUSTOM_PROCESSING",custom_sql 也只会执行固定的一条 SQL,导致其他表的数据无法被清理,旧数据残留。

已尝试方案
1.在 data_save_mode = "DROP_DATA"时 ,日志显示 DataSaveMode 仍为 APPEND_DATA,可能该参数未正确生效。请问data_save_mode = "DROP_DATA",是否需要什么参数配置才能生效。

2.在schema_save_mode = RECREATE_SCHEMA 时,并没有drop ... create , 而是执行create if not exists table1, 此能否实现先drop table 然后再create if not exists, 或者说是否需要什么参数配置才能生效。

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue points to the Doris Sink and the data_save_mode, custom_sql, sink.enable-delete, and schema_save_mode settings; start by tracing how these settings are parsed and applied during multi-table synchronization. Compare the observed DROP_DATA and RECREATE_SCHEMA behavior with the logs, and consider the overwrite behavior complete when each matched target table is cleared before synchronization as requested.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.