apache / apache/hudi

Batch snapshot reads ignore hoodie.file.group.reader.enabled

Open
#19,846 1 comment 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:query-engine priority:medium
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Since #14061, `hoodie.file.group.reader.enabled` is only consulted by the streaming sources (`HoodieStreamSourceV1`/`V2`) and `HoodieIncrSource`. `DefaultSource` routes every batch snapshot and read-optimized query to `HoodieFileGroupReaderBasedFileFormat` regardless of the config, so there is no runtime way to run the same query against the legacy read path.

That gap shows up when a read regression is suspected on the file group reader. On 1.1.x this config was the switch used to compare the two read paths on the same table and cluster; on 1.2.0 and master there is no equivalent, and the config documentation ("Use engine agnostic file group reader if enabled") still reads as if it applies to batch reads.

The legacy relations for snapshot queries are all still in the tree (`BaseFileOnlyRelation`, `MergeOnReadSnapshotRelation`) and still serve metadata-table reads and streaming reads with the config off, so honoring the config for batch snapshot queries again is a routing change only.

Proposal: have `DefaultSource` consult `hoodie.file.group.reader.enabled` for COW snapshot/read-optimized and MOR snapshot queries, leaving the default (config on) behavior unchanged. CDC, incremental and bootstrap queries stay on the file group reader unconditionally, since their legacy relations have been removed.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.