[SUPPORT]Data loss occurs when using bulkinsert
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
Hello,when I use hudi's bulkinsert write type, data loss occurs, regardless of the mor table, cow table or any index type. The amount of data in the source table is 30 million, but when I write to hudi table, about 1000 pieces of data will be missing. And the results of multiple tests show that the amount of data lost is constant. At the same time, it does not change with the table type and index type. Most importantly, the execution logs of spark tasks do not show any error messages. This is very confusing to me, does anyone else have this problem?
**Environment Description**
* Hudi version :0.12.2
* Spark version :3.1.3
* Storage (HDFS/S3/GCS..) :S3
* Running on Docker? (yes/no) :K8S
**Stacktrace**
The execution logs of spark tasks do not show any error messages!
Contributor guide
No contributing guide indexed for this repository
Research direction
No repository file, test, or entry point is named. Start by reproducing the Hudi 0.12.2 bulk_insert job with Spark 3.1.3 on S3/Kubernetes and compare the 30-million-row source with the Hudi output across table and index types. Done means a deterministic reproduction with enough configuration and logs to identify where the approximately 1,000 rows are lost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, spark
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100