apache / apache/hudi

Problem with logical type in precombine field, upsert operation. HUDI 1.1.1, Spark 4.0.2.

Open
#18,610 2 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**

Crashed with:
```
Caused by: java.lang.ClassCastException: class org.apache.avro.generic.GenericData$Fixed cannot be cast to class java.math.BigDecimal (org.apache.avro.generic.GenericData$Fixed is in unnamed module of loader 'app'; java.math.BigDecimal is in module java.base of loader 'bootstrap')
```

**What you expected:**
UPSERT records correctly.

**Steps to reproduce:**
1. Run the example

I think this can probably happen with any LogicalType in Avro terms. **It think** / It **looks** like it works correctly, when precombine field is a part of record key (like: recordkey=id,ts, precombine=ts).

### Environment

**Hudi version:**

1.1.1

**Query engine:** (Spark/Flink/Trino etc)

Spark 4.0.2

**Relevant configs:**

Only source code.

### Logs and Stack Trace

Caused by: java.lang.ClassCastException: class org.apache.avro.generic.GenericData$Fixed cannot be cast to class java.math.BigDecimal (org.apache.avro.generic.GenericData$Fixed is in unnamed module of loader 'app'; java.math.BigDecimal is in module java.base of loader 'bootstrap')
at java.base/java.math.BigDecimal.compareTo(BigDecimal.java:309)
at org.apache.hudi.common.table.read.BufferedRecordMergerFactory.shouldKeepNewerRecord(BufferedRecordMergerFactory.java:493)
at org.apache.hudi.common.table.read.BufferedRecordMergerFactory$EventTimeRecordMerger.finalMerge(BufferedRecordMergerFactory.java:194)
at org.apache.hudi.common.table.read.buffer.FileGroupRecordBuffer.hasNextBaseRecord(FileGroupRecordBuffer.java:242)
at org.apache.hudi.common.table.read.buffer.KeyBasedFileGroupRecordBuffer.hasNextBaseRecord(KeyBasedFileGroupRecordBuffer.java:139)
at org.apache.hudi.common.table.read.buffer.KeyBasedFileGroupRecordBuffer.doHasNext(KeyBasedFileGroupRecordBuffer.java:148)
at org.apache.hudi.common.table.read.buffer.FileGroupRecordBuffer.hasNext(FileGroupRecordBuffer.java:153)
at org.apache.hudi.common.table.read.HoodieFileGroupReader.hasNext(HoodieFileGroupReader.java:246)
at org.apache.hudi.common.table.read.HoodieFileGroupReader$HoodieFileGroupReaderIterator.hasNext(HoodieFileGroupReader.java:333)
at org.apache.hudi.common.util.collection.MappingIterator.hasNext(MappingIterator.java:39)
at org.apache.hudi.io.FileGroupReaderBasedMergeHandle.doMerge(FileGroupReaderBasedMergeHandle.java:271)
at org.apache.hudi.io.IOUtils.runMerge(IOUtils.java:121)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdate(BaseSparkCommitActionExecutor.java:390)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:356)

[HudiBigDecimalBug.java](https://github.com/user-attachments/files/27130212/HudiBigDecimalBug.java)

[pom.xml](https://github.com/user-attachments/files/27130223/pom.xml)
[README.md](https://github.com/user-attachments/files/27130224/README.md)

(sorry, I was not to able to attach whole miniproject as tgz).

Thank you,

David Sauer

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with BufferedRecordMergerFactory.shouldKeepNewerRecord at line 493 and reproduce the failure using HudiBigDecimalBug.java, pom.xml, and README.md. Trace how the logical-type precombine value is represented during the upsert merge, then verify that the example completes without the ClassCastException and upserts records correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.