apache / apache/hudi

Support using event time for timeline

Open
#15,339 0 comments 0 reactions 0 assignees View on GitHub
from-jira priority:high type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Now the instant time in timeline is the proccessing time of a record in hudi. But in some cases, users may need the event time for reading. For example, we import a mysql table to hudi realtimely, but some batch job need to calculate the data for yesterday, they need a snapshot of the mysql table at 00:00, this should be the event time,  not instant time, as it may delay due to slow running or job restarting and so on. So We proposal to support user define using event time as the instant time in timeline.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4622
- Type: New Feature

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the issue description and the linked JIRA ticket HUDI-4622; no source files, tests, or entry points are identified. Trace how Hudi represents processing time in its timeline and determine the required configuration and event-time semantics. Done should include a defined behavior for event-time snapshots and corresponding validation, but the issue does not specify where tests belong.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.