alibaba / alibaba/DataX

对于脏数据提示的改进建议

Open
#852 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

现在对于脏数据配置中有个maxDirtyNumber的配置进行设置,重复数据大于这个值就会报脏数据:
public void collectDirtyRecord(Record dirtyRecord, Throwable t,
String errorMessage) {
int logNum = currentLogNum.getAndIncrement();
if(logNum==0 && t!=null){
LOG.error("", t);
}
if (maxLogNum.intValue() < 0 || currentLogNum.intValue() < maxLogNum.intValue()) {
LOG.error("脏数据: \n"
+ this.formatDirty(dirtyRecord, t, errorMessage));
}

super.collectDirtyRecord(dirtyRecord, t, errorMessage);
}

但是代码中的逻辑只是提示这是脏数据,并不会阻断传输,所以这里报error日志是会误导使用者,以为传输会失败。
其次在测试的时候经常会自己造大量重复数据,这种情况下不需要对脏数据进行提示,建议增加是否判断脏数据的开关配置。

Contributor guide

No contributing guide indexed for this repository

Research direction

Start from the collectDirtyRecord method shown in the issue and trace how maxDirtyNumber is configured and how dirty records are reported. Clarify the desired logging severity and the opt-out configuration with maintainers; done should make the behavior configurable without implying that data transfer failed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.