apache / apache/hudi

Add single-writer validation if concurrency control is not enabled

Open
#15,330 0 comments 0 reactions 0 assignees View on GitHub
from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

In case of misconfiguration or unintended deployment model, where multiple writers write to the same table without concurrency control, the writer should be able to detect that through what's on the storage, e.g., the Hudi timeline, and abort the commit.  If we can enforce a pre-commit check leveraging the timeline (in single-writer case, there should not be new complete/inflight commits this writer is unaware of), an accidental multi-writer case should fail one writer and prevent the table from corrupted.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4599
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Hudi's writer pre-commit path and how the storage timeline exposes complete and inflight commits; the issue names no file or test. Done means a single-writer deployment can detect an unobserved commit and abort one writer before corruption occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.