Clustering w/ two multi-writers could lead to issues
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
if two writers have enabled clustering, each could rollback the clustering that the other writer is currently executing and could lead to unrecoverable issues.
{code:java}
t1 t2
--------------------------------------------------------➝ t
writer1 |-----------------------------|
writer2 |--------------| {code}
lets say writer1 starts a clustering at t1.
and then writer2 starts clustering at time t2. at this time, it will rollback the clustering started at time t1. but writer 1 could still be continuing to execute the clustering.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5174
- Type: Bug
---
## Comments
08/Dec/22 23:02;haox;Hi do we have any updates on this issue?;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no source file or test entry point. Start with JIRA HUDI-5174 and trace the clustering rollback behavior for the overlapping t1/t2 writer scenario described. Done means concurrent clustering no longer lets one writer roll back another writer’s in-progress work, with regression coverage for the two-writer case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100