apache / apache/hudi

[SUPPORT] Order rows with same key before precombine

Open
#11,041 1 comment 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I have an use case where I would like to use hudi. I have to process several inserts, updates and deletes indicated in a file. The file can have lots of rows for the same key and I have to combine it in order using a file. I have developed my own Payload but I can process the rows in order. I have used the options **hoodie.datasource.write.precombine.field** to indicate the precombine field and **hoodie.payload.ordering.field** to order by the same field but it didn't work.
Also, I have tested using repartition and sort functions in the spark code before saving hudi but it didn't work either.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the behavior with the custom Payload and the hoodie.datasource.write.precombine.field and hoodie.payload.ordering.field options. Compare that with the Spark repartition and sort attempt, then trace the relevant Hudi write and precombine entry points. Done means determining whether same-key rows can be processed in file order and documenting or correcting the observed behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.