ADS layer malfunctioned. After restarting the job, the data statistics were incorrect
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Describe the problem you faced
Based on Flink and Hudi's real-time data warehouse, the ADS layer job has malfunctioned, while the ODS and DWD layers have normal job. Restart the ADS layer job one day after the failure. At this point, there are two types of data anomalies. One is that the new data overwrites the old data. For example, the newly calculated March monthly report data will overwrite the old March monthly report; The second type is that new tasks only retrieve data from the most recent commit and do not count the entire amount of data, resulting in new monthly report data being less than the actual value.
How did everyone solve it?
### Environment Description
Hudi version : 0.14.1
Flink version : 1.17
Storage (HDFS/S3/GCS..) : HDFS
Running on Yarn? (yes/no) : yes
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the ADS restart scenario described for Hudi 0.14.1 with Flink 1.17 on HDFS and Yarn, comparing the ADS job with the unaffected ODS and DWD jobs. Determine whether the overwrite and latest-commit-only results share a cause, then document a verified recovery or configuration change and confirm that a restarted job recomputes the complete monthly report.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100