apache / apache/hudi

Fix RawTripTestPayload testcases to use the same payload for table config

Open
#17,139 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 4h
Merged PRs (30d)
112

Description

It was seen that for RawTripTestPayload currently OverwriteWithLatestAvroPayload is set in the table config payload. We should fix the use cases to use RawTripTestPayload itself in the table config payload.
We should also add validation that payload class is same for custom payload merger here [https://github.com/lokeshj1703/hudi/blob/9c6f5306abd71279f5eb32ab5d6ff1bd56533b42/hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java#L101]

refer: [https://github.com/apache/hudi/pull/13600#discussion_r2253901023]

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9689
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the RawTripTestPayload test cases and the custom payload merger handling in hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordMergerFactory.java around the referenced line. Check where OverwriteWithLatestAvroPayload is used in the table configuration and where payload classes are selected. Done means the table configuration uses RawTripTestPayload and validation rejects a different custom payload class.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.