apache / apache/hudi

[SUPPORT] Getting error when writing to different HUDI tables in different threads in same job

Open
#9,344 6 comments 0 reactions 0 assignees View on GitHub
area:concurrency priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

An error occurred while calling o737.save. : org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20230801092005015 at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:75) at org.apache.hudi.table.action.deltacommit.SparkUpsertDeltaCommitActionExecutor.execute(SparkUpsertDeltaCommitActionExecutor.java:45)

**To Reproduce**

Steps to reproduce the behavior:

We are trying an approach where we can upsert 2 Hudi tables at the same time using threadPoolExecutor of Python. The tables are in different folders but in the same S3 bucket. It worked for the initial run but after that we are getting this error. After encountering this error, we tried this property - "ALLOW_MULTI_WRITE_ON_SAME_INSTANT_ENABLE" : "true" but got the same error.

**Expected behavior**

We should be able to upsert into HUDI without any issue.

**Environment Description**

* Hudi version : 0.13.0

* Spark version : 3.3

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : No

**Additional context**

The writer in our case is a python3 code on EC2 (c7g.2xlarge)

**Stacktrace**

PFA
[error log.txt](https://github.com/apache/hudi/files/12241278/error.log.txt)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the attached error log and the stack-trace locations in BaseWriteHelper.java and SparkUpsertDeltaCommitActionExecutor.java. Reproduce concurrent upserts to two Hudi table folders in one S3 bucket using Python's threadPoolExecutor, including ALLOW_MULTI_WRITE_ON_SAME_INSTANT_ENABLE, and compare the result with sequential writes. Done means both concurrent table upserts complete successfully without the HoodieUpsertException.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, python, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.