apache / apache/hudi

Add guard rails to fail fast on multiple writers (excluding async table services) for a given table

Open
#15,708 0 comments 0 reactions 0 assignees View on GitHub
area:writer from-jira priority:critical type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

With deltastreamer continuous mode and async table services, we auto enable OCC and lazy rollback bcoz, there is only a single writer to metadata table. 

So, if the table had multiple concurrent writers by mistake (concurrent spark ds writer), there is a chance we let them proceed. Some users prefer to add guard rails around this. 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5598
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing deltastreamer continuous mode and the async table services behavior described in the issue, including how OCC and lazy rollback are enabled. Define done as failing fast when multiple concurrent writers target a table while preserving the async table services exception; no file or test is named, so locating the relevant implementation and coverage is part of the work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.