apache / apache/hudi

[SUPPORT]: Data loss with Concurrent operations on Hudi MOR

Open
#9,674 11 comments 0 reactions 0 assignees View on GitHub
area:concurrency area:table-service issue:data-loss priority:critical
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**
While running two concurrent operations on Hudi MOR Table , we are facing data loss.

**To Reproduce**

Steps to reproduce the behavior:
1. Run upsert at 10:30 AM on Husi MOR Table
2. Run cleaner utility on same MOR table 10:31 AM while upsert is still going on

**Note**: Hoodie Cleaner is running as a separate process
Hudi MOR Table configuration:
```
{
'hoodie.table.name': asset,
'hoodie.datasource.write.recordkey.field': id,
'hoodie.datasource.write.table.name': asset,
'hoodie.upsert.shuffle.parallelism': 400,
'hoodie.keep.max.commits': 50,
'hoodie.keep.min.commits': 49,
'hoodie.compact.inline.max.delta.commits': 6,
'hoodie.clean.automatic': 'false',
'hoodie.clean.async': 'false'
}
```
Cleaner Utility:
```
/usr/local/bin/spark-submit
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
--conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.InstanceProfileCredentialsProvider,com.amazonaws.auth.DefaultAWSCredentialsProviderChain
--conf spark.hadoop.fs.AbstractFileSystem.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
--conf spark.jars.packages=org.apache.spark:spark-avro_2.12:3.0.1,org.apache.hadoop:hadoop-aws:3.2.2,com.amazonaws:aws-java-sdk-bundle:1.12.180,org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.0
--class org.apache.hudi.utilities.HoodieCleaner /home/ubuntu/hudi-utilities-bundle.jar
--target-base-path s3a://bucket_name/table_path/reference/
--hoodie-conf hoodie.cleaner.policy=KEEP_LATEST_COMMITS
--hoodie-conf hoodie.keep.max.commits=50
--hoodie-conf hoodie.keep.min.commits=49
--hoodie-conf hoodie.cleaner.commits.retained=48
--hoodie-conf hoodie.cleaner.parallelism=400
```
**Expected behavior**

All data should be available in Hudi MOR Table.

**Environment Description**

* Hudi version : 0.13.0

* Spark version : 3.3.1

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : No

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the org.apache.hudi.utilities.HoodieCleaner entry point and the reported MOR-table configuration, then reproduce an upsert running concurrently with the cleaner on S3. Compare the retained records with the expected data after both operations complete; done means no data is lost.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spark
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.