apache / apache/hudi

[SUPPORT]Data loss occurs when using bulkinsert

Open
#9,748 21 comments 0 reactions 0 assignees View on GitHub
area:schema area:writer issue:data-loss priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**
Hello,when I use hudi's bulkinsert write type, data loss occurs, regardless of the mor table, cow table or any index type. The amount of data in the source table is 30 million, but when I write to hudi table, about 1000 pieces of data will be missing. And the results of multiple tests show that the amount of data lost is constant. At the same time, it does not change with the table type and index type. Most importantly, the execution logs of spark tasks do not show any error messages. This is very confusing to me, does anyone else have this problem?

**Environment Description**

* Hudi version :0.12.2

* Spark version :3.1.3

* Storage (HDFS/S3/GCS..) :S3

* Running on Docker? (yes/no) :K8S

**Stacktrace**

The execution logs of spark tasks do not show any error messages!

Contributor guide

No contributing guide indexed for this repository

Research direction

No repository file, test, or entry point is named. Start by reproducing the Hudi 0.12.2 bulk_insert job with Spark 3.1.3 on S3/Kubernetes and compare the 30-million-row source with the Hudi output across table and index types. Done means a deterministic reproduction with enough configuration and logs to identify where the approximately 1,000 rows are lost.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spark
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.