Add guard rails to fail fast on multiple writers (excluding async table services) for a given table
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
With deltastreamer continuous mode and async table services, we auto enable OCC and lazy rollback bcoz, there is only a single writer to metadata table.
So, if the table had multiple concurrent writers by mistake (concurrent spark ds writer), there is a chance we let them proceed. Some users prefer to add guard rails around this.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5598
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing deltastreamer continuous mode and the async table services behavior described in the issue, including how OCC and lazy rollback are enabled. Define done as failing fast when multiple concurrent writers target a table while preserving the async table services exception; no file or test is named, so locating the relevant implementation and coverage is part of the work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100