apache / apache/hudi

Use state transition times for SimpleConcurrentFileWritesConflictResolutionStrategy and other enhancements.

Open
#16,015 0 comments 0 reactions 0 assignees View on GitHub
from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Couple of enhancements we can make to Multiwriter conflict resolution strategies.
* Similar to how git rebase works, writers when creating a requested commit should track the baseInstant on which they are executing, this will be the last completed commit. Note, last completed commit can have commit timestamp less than other completed commits in the timeline. Here, baseInstantTime is based on the state transition times which is the last commit written to the timeline. Conflict resolution strategies should happen based on this commit time. We can also avoid the race condition by acquiring lock during requested file creation.
* Similar to IngestionPrimaryWriterBasedConflictResolutionStrategy class, we should use state transition times to find the conflicts. By using state transition times the conflict resolution works similar to git rebase.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6353
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading SimpleConcurrentFileWritesConflictResolutionStrategy and IngestionPrimaryWriterBasedConflictResolutionStrategy, then trace how requested commits and state transition times are represented. Compare the existing conflict paths with the linked HUDI-6353 requirements; done means the affected strategies consistently use state transition times and the requested-commit race is addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.