apache / apache/hudi

[DISCUSSION] Should we treat `COMMIT_TIME_ORDERING` as a special case of `EVENT_TIME_ORDERING` ?

Open
#12,997 4 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

From the current code structure, we would treat the merge policy of `COMMIT_TIME_ORDERING` as a separate logic, but from a business perspective, should we treat it as a special case of `EVENT_TIME_ORDERING` where the ORDERING VALUE for each record is the same? For example, right now it's represented by `int: 0`. This way we don't need to maintain two merge policies, we default to `EVENT_TIME_ORDERING`, the same record_key merge policy is handled in the order of `transaction_time` and `event_time`. This will help us in the future to maintain the code and deal with the various merge problems we have encountered.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the existing COMMIT_TIME_ORDERING and EVENT_TIME_ORDERING merge-policy paths, including how transaction_time and event_time determine record ordering. Clarify whether the policies can share one implementation and define the required behavior before identifying affected tests and validating the change.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.