apache / apache/hudi

HoodieIndexUtils.filterKeysFromFile hardcodes AVRO fallback, ignoring configured record merger type

Open
#18,496 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Describe the problem

In [`HoodieIndexUtils.filterKeysFromFile`](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java#L253) the record type used to open the base file reader for key filtering is computed as:

```java
HoodieRecord.HoodieRecordType recordType =
HoodieFileFormat.fromFileExtension(FSUtils.getFileExtension(filePath.toString()))
.resolveRecordType(HoodieRecord.HoodieRecordType.AVRO);
```

`resolveRecordType` only overrides the fallback when the file format **requires** a specific record type (currently only Lance). For all other formats (Parquet, ORC, HFile) the AVRO fallback is hardcoded and ignores the merger's configured record type. This predates the Lance work (#18375).

### Impact

If a user configures a SPARK-type merger for a Parquet table, key filtering during index lookup will still open an AVRO reader, incurring unnecessary Avro<->InternalRow conversions in the key-filter code path.

This is not a correctness bug — `filterRowKeys` only reads record keys — but it is an efficiency/consistency gap flagged during review of #18375 (cf. [review thread](https://github.com/apache/hudi/pull/18375#discussion_r3036297458)).

### Proposed fix

Thread the configured merger's record type into `filterKeysFromFile` (or read it from config via an argument) and pass it as the fallback to `resolveRecordType` instead of hardcoding `AVRO`.

### Context

- Raised by @rahil-c in https://github.com/apache/hudi/pull/18375#discussion_r3069697589
- File: `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java` around line 254 (the `// TODO: AVRO fallback` comment).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java at filterKeysFromFile and inspect its callers and the review context from #18375. Trace how the configured merger record type is available, then verify that it is used as the fallback while formats requiring a specific type, such as Lance, still override it. Done means key filtering no longer hardcodes AVRO for configurable formats.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.