apache / apache/hudi

[SUPPORT] Dataloss in FlinkCDC into Hudi without any exception or other infomation

Open
#10,542 18 comments 0 reactions 0 assignees View on GitHub
area:cdc engine:flink issue:data-loss issue:function-regression
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

**Describe the problem you faced**

When we use FlinkCDC into HUDI 0.14.0/0.14.1,would occur dataloss such as deleted data in mysql but can find the data in Hudi,or insert data to mysql but not find data in Hudi,more important not produce any exception for us to dig cause root.
sences:
1. Flink write to Hudi without compaction
2. Flink write to Hudi with compaction online
3. Flink write to Hudi with compaction offline
above 3 conditions all can occur the problem

**To Reproduce**

Steps to reproduce the behavior:

1. Flink write to hudi with Billions data
2. then check data with mysql

**Expected behavior**

all data should consistent with mysql

**Environment Description**

* Flink version : 1.13.2

* Hudi version : 0.14.0/0.14.1

* Spark version : 3.2.0

* Hive version : 1.1.0

* Hadoop version : 3.2.1

* Storage (HDFS/S3/GCS..) : HDFS

* Running on Docker? (yes/no) :no

**Additional context**

Add any other context about the problem here.

**Stacktrace**

Not any stacktrace to dig it,flink and compaction job is fine ant the time

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the reported Flink-to-Hudi loss in the three listed compaction modes using the stated Flink, Hudi, Spark, Hive, Hadoop, and HDFS versions. Compare MySQL and Hudi contents after the billion-row run and capture enough job or compaction output to define the discrepancy; done means identifying a reproducible inconsistency or producing evidence that the data remains consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, spark
Domain
data-engineering, databases, stream-processing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.