apache / apache/doris-flink-connector

[Enhancement] Skip CDC delete events early when sink.enable-delete=false to reduce stream_load overhead

Open
#644 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
385
Forks
279
Avg merge
2d 2h
Merged PRs (30d)
6

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Description

当前删除事件被转换为 __DORIS_DELETE_SIGN=0 而不是真正过滤掉
造成无效的序列化、网络传输和 stream_load 处理

### Solution

在 ParsingProcessFunction 或 Serializer 层面提前过滤删除事件
返回 null 跳过序列化过程
添加测试确保不会产生 __DORIS_DELETE_SIGN=0 的记录

✅ 影响
减少网络流量
降低 Doris FE/BE 压力
提升 CDC 数据流效率

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Locate ParsingProcessFunction and the Serializer, then trace how CDC delete events are handled when sink.enable-delete=false. Add a focused test for the disabled-delete path; done means delete events are skipped before serialization and no __DORIS_DELETE_SIGN=0 record is produced.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.