Problem with logical type in precombine field, upsert operation. HUDI 1.1.1, Spark 4.0.2.
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:**
Crashed with:
```
Caused by: java.lang.ClassCastException: class org.apache.avro.generic.GenericData$Fixed cannot be cast to class java.math.BigDecimal (org.apache.avro.generic.GenericData$Fixed is in unnamed module of loader 'app'; java.math.BigDecimal is in module java.base of loader 'bootstrap')
```
**What you expected:**
UPSERT records correctly.
**Steps to reproduce:**
1. Run the example
I think this can probably happen with any LogicalType in Avro terms. **It think** / It **looks** like it works correctly, when precombine field is a part of record key (like: recordkey=id,ts, precombine=ts).
### Environment
**Hudi version:**
1.1.1
**Query engine:** (Spark/Flink/Trino etc)
Spark 4.0.2
**Relevant configs:**
Only source code.
### Logs and Stack Trace
Caused by: java.lang.ClassCastException: class org.apache.avro.generic.GenericData$Fixed cannot be cast to class java.math.BigDecimal (org.apache.avro.generic.GenericData$Fixed is in unnamed module of loader 'app'; java.math.BigDecimal is in module java.base of loader 'bootstrap')
at java.base/java.math.BigDecimal.compareTo(BigDecimal.java:309)
at org.apache.hudi.common.table.read.BufferedRecordMergerFactory.shouldKeepNewerRecord(BufferedRecordMergerFactory.java:493)
at org.apache.hudi.common.table.read.BufferedRecordMergerFactory$EventTimeRecordMerger.finalMerge(BufferedRecordMergerFactory.java:194)
at org.apache.hudi.common.table.read.buffer.FileGroupRecordBuffer.hasNextBaseRecord(FileGroupRecordBuffer.java:242)
at org.apache.hudi.common.table.read.buffer.KeyBasedFileGroupRecordBuffer.hasNextBaseRecord(KeyBasedFileGroupRecordBuffer.java:139)
at org.apache.hudi.common.table.read.buffer.KeyBasedFileGroupRecordBuffer.doHasNext(KeyBasedFileGroupRecordBuffer.java:148)
at org.apache.hudi.common.table.read.buffer.FileGroupRecordBuffer.hasNext(FileGroupRecordBuffer.java:153)
at org.apache.hudi.common.table.read.HoodieFileGroupReader.hasNext(HoodieFileGroupReader.java:246)
at org.apache.hudi.common.table.read.HoodieFileGroupReader$HoodieFileGroupReaderIterator.hasNext(HoodieFileGroupReader.java:333)
at org.apache.hudi.common.util.collection.MappingIterator.hasNext(MappingIterator.java:39)
at org.apache.hudi.io.FileGroupReaderBasedMergeHandle.doMerge(FileGroupReaderBasedMergeHandle.java:271)
at org.apache.hudi.io.IOUtils.runMerge(IOUtils.java:121)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdate(BaseSparkCommitActionExecutor.java:390)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:356)
[HudiBigDecimalBug.java](https://github.com/user-attachments/files/27130212/HudiBigDecimalBug.java)
[pom.xml](https://github.com/user-attachments/files/27130223/pom.xml)
[README.md](https://github.com/user-attachments/files/27130224/README.md)
(sorry, I was not to able to attach whole miniproject as tgz).
Thank you,
David Sauer
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with BufferedRecordMergerFactory.shouldKeepNewerRecord at line 493 and reproduce the failure using HudiBigDecimalBug.java, pom.xml, and README.md. Trace how the logical-type precombine value is represented during the upsert merge, then verify that the example completes without the ClassCastException and upserts records correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100