apache / apache/doris-flink-connector
[Enhancement] Skip CDC delete events early when sink.enable-delete=false to reduce stream_load overhead
- Dominant language
- Java
- Stars
- 385
- Forks
- 279
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 6
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
当前删除事件被转换为 __DORIS_DELETE_SIGN=0 而不是真正过滤掉
造成无效的序列化、网络传输和 stream_load 处理
### Solution
在 ParsingProcessFunction 或 Serializer 层面提前过滤删除事件
返回 null 跳过序列化过程
添加测试确保不会产生 __DORIS_DELETE_SIGN=0 的记录
✅ 影响
减少网络流量
降低 Doris FE/BE 压力
提升 CDC 数据流效率
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Locate ParsingProcessFunction and the Serializer, then trace how CDC delete events are handled when sink.enable-delete=false. Add a focused test for the disabled-delete path; done means delete events are skipped before serialization and no __DORIS_DELETE_SIGN=0 record is produced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100