apache / apache/hudi

ADS layer malfunctioned. After restarting the job, the data statistics were incorrect

Open
#11,017 2 comments 0 reactions 0 assignees View on GitHub
area:incr-processing area:streaming engine:flink type:bug type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Describe the problem you faced
Based on Flink and Hudi's real-time data warehouse, the ADS layer job has malfunctioned, while the ODS and DWD layers have normal job. Restart the ADS layer job one day after the failure. At this point, there are two types of data anomalies. One is that the new data overwrites the old data. For example, the newly calculated March monthly report data will overwrite the old March monthly report; The second type is that new tasks only retrieve data from the most recent commit and do not count the entire amount of data, resulting in new monthly report data being less than the actual value.
How did everyone solve it?

### Environment Description
Hudi version : 0.14.1
Flink version : 1.17
Storage (HDFS/S3/GCS..) : HDFS
Running on Yarn? (yes/no) : yes

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the ADS restart scenario described for Hudi 0.14.1 with Flink 1.17 on HDFS and Yarn, comparing the ADS job with the unaffected ODS and DWD jobs. Determine whether the overwrite and latest-commit-only results share a cause, then document a verified recovery or configuration change and confirm that a restarted job recomputes the complete monthly report.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.