apache / apache/hudi

fix(metadata): ArrayIndexOutOfBoundsException when 1.x reader merges MDT records written by older Hudi versions

Open
#19,559 0 comments 0 reactions 1 assignee Claimed by @lokeshj1703 View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Problem

When a Hudi 1.x reader reads a Metadata Table (MDT) that was written by an older Hudi release, merging log-file delta records against base-file records throws an `ArrayIndexOutOfBoundsException`:

```
java.lang.ArrayIndexOutOfBoundsException: Index 6 out of bounds for length 6
at org.apache.avro.generic.GenericData$Record.put(GenericData.java:275)
at org.apache.hudi.metadata.HoodieMetadataPayload.getInsertValue(HoodieMetadataPayload.java)
at org.apache.hudi.metadata.HoodieMetadataPayload.combineAndGetUpdateValue(HoodieMetadataPayload.java)
at org.apache.hudi.common.model.HoodieAvroRecordMerger.merge(HoodieAvroRecordMerger.java)
```

## Root Cause

`SecondaryIndexMetadata` was added to `HoodieMetadataRecord` in a later Hudi release, growing the schema from 6 to 7 fields. MDT log blocks written by older releases carry the 6-field writer schema in their block header.

When merging, `HoodieAvroRecordMerger` retrieves the writer schema from the buffered record (`getSchemaFromBufferRecord`) and passes it to `HoodieMetadataPayload.getInsertValue(schema)`. Inside `getInsertValue`, the fast path uses an object-identity check:

```java
if (schema == null || schema == HOODIE_METADATA_AVRO_SCHEMA) {
// safe path – return HoodieMetadataRecord directly
} else {
// assumes HOODIE_META_COLUMNS are prepended; uses KEY_FIELD_OFFSET=5, TYPE_FIELD_OFFSET=6
record.put(TYPE_FIELD_OFFSET, type); // index 6 on a 6-field schema → AIOOBE
}
```

The old 6-field schema is a different `Schema` object from `HOODIE_METADATA_AVRO_SCHEMA` (7 fields), so it fails the identity check and falls into the `else` branch, which is designed for schemas with `HOODIE_META_COLUMNS` prepended. Since the old schema has only 6 fields (indices 0–5), writing to index 6 throws `ArrayIndexOutOfBoundsException`.

## Fix

Add a helper `isHoodieMetadataRecordSchema(Schema)` that recognises any version of the `HoodieMetadataRecord` schema by Avro record name and namespace. Route these older-version schemas to the same fast path as the current schema, returning a `HoodieMetadataRecord` with the current 7-field schema (missing `SecondaryIndexMetadata` is `null`).

This preserves backward compatibility: tables written by older Hudi readers can be read safely by a 1.x reader without requiring a table upgrade.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.