apache / apache/hudi

Unify Creation of Timestamps and Persistence to Timeline

Open
#16,999 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:epic
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently we take a lock to generate timestamps which ensures that the timestamps are generated in a strictly increasing order. This helps us reason about the order of the commits but the timeline is not necessarily updated when these timestamps when they are created. This leads to inconsistencies when dealing with multiple writers since the other writer may not have the knowledge of this new timestamp. For the other writer to have this knowledge, we need to persist the instant to the timeline.

On top of this, we want to minimize the amount of time where the table is locked while still ensuring correctness and preventing any opportunities for corruption of the table or its metadata. This will help us better support multiple writers to the same table going forward.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9384
- Type: Epic

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the linked JIRA issue HUDI-9384 and the issue description, focusing on timestamp creation, timeline persistence, multiple writers, and table locking. The payload names no files, tests, or entry points; completion would require a defined design and implementation that preserves correctness while reducing lock duration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.