Detect HoodieMerge type with engine type automatically instead of the default avro-based one
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
The same table be read by Spark or Hive for e.g I would love for such queries to be able to read and merge using the engine' POJO (Spark -> Row, Hive -> ArrayWritable), if the user chooses to write such a "merge" implementation.
For e.g consider this API
{code:java}
Option combineAndGetUpdateValue(HoodieRecord older, HoodieRecord newer, Schema schema, Properties props) {
...
} {code}
when writing from Spark we would use `HoodieSparkRecord` and the merge class would be `HoodieSparkMerge` in the table properties? and if we now read with Hive, it falls back to avro somehow?
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4301
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the HoodieRecord combineAndGetUpdateValue API and the table properties that select the merge implementation. Compare the Spark and Hive read/write paths, then define how the engine-specific merge type is detected and how Avro remains the fallback; done means both engines can read and merge compatible tables without manual default selection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100