对于脏数据提示的改进建议
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
现在对于脏数据配置中有个maxDirtyNumber的配置进行设置,重复数据大于这个值就会报脏数据:
public void collectDirtyRecord(Record dirtyRecord, Throwable t,
String errorMessage) {
int logNum = currentLogNum.getAndIncrement();
if(logNum==0 && t!=null){
LOG.error("", t);
}
if (maxLogNum.intValue() < 0 || currentLogNum.intValue() < maxLogNum.intValue()) {
LOG.error("脏数据: \n"
+ this.formatDirty(dirtyRecord, t, errorMessage));
}
super.collectDirtyRecord(dirtyRecord, t, errorMessage);
}
但是代码中的逻辑只是提示这是脏数据,并不会阻断传输,所以这里报error日志是会误导使用者,以为传输会失败。
其次在测试的时候经常会自己造大量重复数据,这种情况下不需要对脏数据进行提示,建议增加是否判断脏数据的开关配置。
Contributor guide
No contributing guide indexed for this repository
Research direction
Start from the collectDirtyRecord method shown in the issue and trace how maxDirtyNumber is configured and how dirty records are reported. Clarify the desired logging severity and the opt-out configuration with maintainers; done should make the behavior configurable without implying that data transfer failed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100