[SUPPORT] Order rows with same key before precombine
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I have an use case where I would like to use hudi. I have to process several inserts, updates and deletes indicated in a file. The file can have lots of rows for the same key and I have to combine it in order using a file. I have developed my own Payload but I can process the rows in order. I have used the options **hoodie.datasource.write.precombine.field** to indicate the precombine field and **hoodie.payload.ordering.field** to order by the same field but it didn't work.
Also, I have tested using repartition and sort functions in the spark code before saving hudi but it didn't work either.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the behavior with the custom Payload and the hoodie.datasource.write.precombine.field and hoodie.payload.ordering.field options. Compare that with the Spark repartition and sort attempt, then trace the relevant Hudi write and precombine entry points. Done means determining whether same-key rows can be processed in file order and documenting or correcting the observed behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100