apache / apache/hudi

Deprecate `hoodie.compaction.payload.class` and re-use hoodie.datasource.write.payload.class

Open
#15,438 1 comment 0 reactions 0 assignees View on GitHub
area:config area:table-service area:writer component:compaction from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

We have two configs for payload which is causing confusion. 

 

hoodie.datasource.write.payload.class

and hoodie.compaction.payload.class

2nd is a table config, which will be used by MOR snapshot query to merge base with log records. 

 

From the code, looks like hoodie.compaction.payload.class is overwritten w/ whats set for hoodie.datasource.write.payload.class. So, hoodie.datasource.write.payload.class takes precedence. but lets see if we can deprecate one and use 1 config across the board. 

 

 

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4863
- Type: Improvement

---

## Comments

03/Nov/22 21:09;jonvex;remove the config from the guide and the configuration reference;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Search the codebase for hoodie.compaction.payload.class and hoodie.datasource.write.payload.class, then inspect their uses and the configuration guide and configuration reference. Done means the redundant compaction setting is deprecated or removed, one payload configuration is used consistently, and the affected documentation is updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.