apache / apache/hudi

Detect HoodieMerge type with engine type automatically instead of the default avro-based one

Open
#15,247 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

The same table be read by Spark or Hive for e.g I would love for such queries to be able to read and merge using the engine' POJO (Spark -> Row, Hive -> ArrayWritable), if the user chooses to write such a "merge" implementation. 

For e.g consider this API
{code:java}
Option combineAndGetUpdateValue(HoodieRecord older, HoodieRecord newer, Schema schema, Properties props) {
...
} {code}
when writing from Spark we would use `HoodieSparkRecord` and the merge class would be `HoodieSparkMerge` in the table properties? and if we now read with Hive, it falls back to avro somehow? 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4301
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the HoodieRecord combineAndGetUpdateValue API and the table properties that select the merge implementation. Compare the Spark and Hive read/write paths, then define how the engine-specific merge type is detected and how Avro remains the fallback; done means both engines can read and merge compatible tables without manual default selection.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.