apache / apache/hudi

Unify the logic of intra-partition upsert and cross-partition upsert in flink state index.

Open
#15,944 2 comments 0 reactions 0 assignees View on GitHub
area:index from-jira priority:medium status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Now when partitioning upsert, according to {{precombine.field}} parameter, keep the record with the largest value after upserting.

This is widely used to solve the case of out-of-order data, by setting the {{precombine.field}} to the event time to keep records with the largest event time.

However, when using the FLINK_STATE index type, if cross-partition occurs, the precombine.field parameter will not fully take effect.

In the case of cross-partitioning, the current logic uses data that arrives later, even if the event time is smaller.

It may be necessary to unify the logic of intra-partition upsert and cross-partition upsert, which is convenient for users to understand and use.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6188
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the FLINK_STATE index's intra-partition and cross-partition upsert paths, then compare how precombine.field is applied in each case. Reproduce an out-of-order cross-partition update and verify that the record with the largest event-time value is retained, matching intra-partition behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.