apache / apache/hudi

Make merger and payload class be consisent

Open
#16,787 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

{{`HoodieRecordUtil.createRecordMerger`}} falls back to `{{{}HoodieAvroRecordMerger`{}}} without considering merger mode. However, `{{{}HoodieTableConfig.inferCorrectMergingBehavior`{}}} behaviors differently. That is, if the merge strategy id is given, it could use `{{{}Commit time based`{}}} payload class, which may contradict with the merger selection logic of `{{{}HoodieRecordUtil`{}}} . What is why sometimes we see there are inconsistent merging behavior. Since the behavior of `{{{}HoodieRecordUtil.createRecordMerger`{}}} affects MDT behavior, we need to update the `{{{}HoodieTableConfig.inferCorrectMergingBehavior`{}}} accordingly.

 

This issue may not impact broadly since this is a corner case.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8876
- Type: Bug
- Affects version(s):
- 1.2.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing HoodieRecordUtil.createRecordMerger with HoodieTableConfig.inferCorrectMergingBehavior, focusing on the fallback and merge strategy id described in the issue. Trace how the selected merger and payload class affect MDT behavior; done means both entry points select consistent merging behavior for the cited corner case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.