Unify Creation of Timestamps and Persistence to Timeline
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently we take a lock to generate timestamps which ensures that the timestamps are generated in a strictly increasing order. This helps us reason about the order of the commits but the timeline is not necessarily updated when these timestamps when they are created. This leads to inconsistencies when dealing with multiple writers since the other writer may not have the knowledge of this new timestamp. For the other writer to have this knowledge, we need to persist the instant to the timeline.
On top of this, we want to minimize the amount of time where the table is locked while still ensuring correctness and preventing any opportunities for corruption of the table or its metadata. This will help us better support multiple writers to the same table going forward.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9384
- Type: Epic
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the linked JIRA issue HUDI-9384 and the issue description, focusing on timestamp creation, timeline persistence, multiple writers, and table locking. The payload names no files, tests, or entry points; completion would require a defined design and implementation that preserves correctness while reducing lock duration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100