apache / apache/hudi

Clustering w/ two multi-writers could lead to issues

Open
#15,550 1 comment 0 reactions 0 assignees View on GitHub
area:table-service component:clustering from-jira priority:critical type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

if two writers have enabled clustering, each could rollback the clustering that the other writer is currently executing and could lead to unrecoverable issues. 

 

 
{code:java}
t1 t2
--------------------------------------------------------➝ t
writer1 |-----------------------------|
writer2 |--------------| {code}
lets say writer1 starts a clustering at t1. 

and then writer2 starts clustering at time t2. at this time, it will rollback the clustering started at time t1. but writer 1 could still be continuing to execute the clustering. 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5174
- Type: Bug

---

## Comments

08/Dec/22 23:02;haox;Hi do we have any updates on this issue?;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no source file or test entry point. Start with JIRA HUDI-5174 and trace the clustering rollback behavior for the overlapping t1/t2 writer scenario described. Done means concurrent clustering no longer lets one writer roll back another writer’s in-progress work, with regression coverage for the two-writer case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.