[SUPPORT] Dataloss in FlinkCDC into Hudi without any exception or other infomation
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
**Describe the problem you faced**
When we use FlinkCDC into HUDI 0.14.0/0.14.1,would occur dataloss such as deleted data in mysql but can find the data in Hudi,or insert data to mysql but not find data in Hudi,more important not produce any exception for us to dig cause root.
sences:
1. Flink write to Hudi without compaction
2. Flink write to Hudi with compaction online
3. Flink write to Hudi with compaction offline
above 3 conditions all can occur the problem
**To Reproduce**
Steps to reproduce the behavior:
1. Flink write to hudi with Billions data
2. then check data with mysql
**Expected behavior**
all data should consistent with mysql
**Environment Description**
* Flink version : 1.13.2
* Hudi version : 0.14.0/0.14.1
* Spark version : 3.2.0
* Hive version : 1.1.0
* Hadoop version : 3.2.1
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) :no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
Not any stacktrace to dig it,flink and compaction job is fine ant the time
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the reported Flink-to-Hudi loss in the three listed compaction modes using the stated Flink, Hudi, Spark, Hive, Hadoop, and HDFS versions. Compare MySQL and Hudi contents after the billion-row run and capture enough job or compaction output to define the discrepancy; done means identifying a reproducible inconsistency or producing evidence that the data remains consistent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, spark
- Domain
- data-engineering, databases, stream-processing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100