Unify the logic of intra-partition upsert and cross-partition upsert in flink state index.
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Now when partitioning upsert, according to {{precombine.field}} parameter, keep the record with the largest value after upserting.
This is widely used to solve the case of out-of-order data, by setting the {{precombine.field}} to the event time to keep records with the largest event time.
However, when using the FLINK_STATE index type, if cross-partition occurs, the precombine.field parameter will not fully take effect.
In the case of cross-partitioning, the current logic uses data that arrives later, even if the event time is smaller.
It may be necessary to unify the logic of intra-partition upsert and cross-partition upsert, which is convenient for users to understand and use.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6188
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the FLINK_STATE index's intra-partition and cross-partition upsert paths, then compare how precombine.field is applied in each case. Reproduce an out-of-order cross-partition update and verify that the record with the largest event-time value is retained, matching intra-partition behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100