fix(metadata): ArrayIndexOutOfBoundsException when 1.x reader merges MDT records written by older Hudi versions
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
## Problem
When a Hudi 1.x reader reads a Metadata Table (MDT) that was written by an older Hudi release, merging log-file delta records against base-file records throws an `ArrayIndexOutOfBoundsException`:
```
java.lang.ArrayIndexOutOfBoundsException: Index 6 out of bounds for length 6
at org.apache.avro.generic.GenericData$Record.put(GenericData.java:275)
at org.apache.hudi.metadata.HoodieMetadataPayload.getInsertValue(HoodieMetadataPayload.java)
at org.apache.hudi.metadata.HoodieMetadataPayload.combineAndGetUpdateValue(HoodieMetadataPayload.java)
at org.apache.hudi.common.model.HoodieAvroRecordMerger.merge(HoodieAvroRecordMerger.java)
```
## Root Cause
`SecondaryIndexMetadata` was added to `HoodieMetadataRecord` in a later Hudi release, growing the schema from 6 to 7 fields. MDT log blocks written by older releases carry the 6-field writer schema in their block header.
When merging, `HoodieAvroRecordMerger` retrieves the writer schema from the buffered record (`getSchemaFromBufferRecord`) and passes it to `HoodieMetadataPayload.getInsertValue(schema)`. Inside `getInsertValue`, the fast path uses an object-identity check:
```java
if (schema == null || schema == HOODIE_METADATA_AVRO_SCHEMA) {
// safe path – return HoodieMetadataRecord directly
} else {
// assumes HOODIE_META_COLUMNS are prepended; uses KEY_FIELD_OFFSET=5, TYPE_FIELD_OFFSET=6
record.put(TYPE_FIELD_OFFSET, type); // index 6 on a 6-field schema → AIOOBE
}
```
The old 6-field schema is a different `Schema` object from `HOODIE_METADATA_AVRO_SCHEMA` (7 fields), so it fails the identity check and falls into the `else` branch, which is designed for schemas with `HOODIE_META_COLUMNS` prepended. Since the old schema has only 6 fields (indices 0–5), writing to index 6 throws `ArrayIndexOutOfBoundsException`.
## Fix
Add a helper `isHoodieMetadataRecordSchema(Schema)` that recognises any version of the `HoodieMetadataRecord` schema by Avro record name and namespace. Route these older-version schemas to the same fast path as the current schema, returning a `HoodieMetadataRecord` with the current 7-field schema (missing `SecondaryIndexMetadata` is `null`).
This preserves backward compatibility: tables written by older Hudi readers can be read safely by a 1.x reader without requiring a table upgrade.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.