HoodieAvroRecord serves the payload's default ordering value, causing ClassCastException on merge
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
`HoodieAvroRecord.doGetOrderingValue` returns the payload's ordering value unconditionally. A
payload built through `HoodieRecordUtils.loadPayload(String, GenericRecord)`, the overload used
whenever a record is created without an explicit ordering value, defaults to
`OrderingValues.getDefault()`, which is `Integer` 0.
Its sibling `HoodieAvroIndexedRecord.doGetOrderingValue` instead reads the ordering fields off the
record, so the two Avro record representations disagree on both the ordering value and its type for
the same input.
When such a record reaches the merge path, its `Integer` 0 is compared against the base record's
ordering value read from storage, which is typed by the ordering field. For any ordering column
that is not an `int`, this throws:
```
java.lang.ClassCastException: class java.lang.Long cannot be cast to class java.lang.Integer
at java.base/java.lang.Integer.compareTo(Integer.java:71)
at org.apache.hudi.common.table.read.BufferedRecordMergerFactory.shouldKeepNewerRecord(BufferedRecordMergerFactory.java:515)
at org.apache.hudi.common.table.read.BufferedRecordMergerFactory$EventTimeRecordMerger.finalMerge(BufferedRecordMergerFactory.java:194)
at org.apache.hudi.common.table.read.buffer.FileGroupRecordBuffer.hasNextBaseRecord(FileGroupRecordBuffer.java:242)
at org.apache.hudi.io.FileGroupReaderBasedMergeHandle.doMerge(FileGroupReaderBasedMergeHandle.java:274)
```
**How to reproduce.** A copy on write table with a `bigint` precombine field, updated through a
prepped Spark SQL write (`UPDATE` / `MERGE INTO`, which set `_hoodie.spark.sql.writes.prepped`).
Those writes leave `shouldCombine` false, so `HoodieCreateRecordUtils` takes the
`createHoodieRecord` overload that carries no ordering value. Every updated row fails.
The same is reachable outside the prepped path: `HoodieCreateRecordUtils` computes the ordering
value with `getNestedFieldVal(..., false, ...)`, which returns `null` for a null field value, and
`loadPayload(clazz, record, null)` then falls through to the same defaulting constructor.
`HoodieAvroIndexedRecord` is unaffected because it derives the value from the record.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with HoodieAvroRecord.doGetOrderingValue and compare it with HoodieAvroIndexedRecord.doGetOrderingValue. Trace payload creation through HoodieRecordUtils.loadPayload and HoodieCreateRecordUtils, then reproduce the prepped Spark SQL update with a bigint precombine field. Done means the merge path handles records without an explicit ordering value without a ClassCastException and the two Avro representations agree.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100