apache / apache/hudi

[FEATURE] hudi-trino: resolve parquet columns through the internal schema for full schema evolution (hoodie.schema.on.read.enable)

Open
#19,381 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Sub-task of #18780 (RFC-105 Trino connector migration).

### Context

The connector resolves parquet columns by NAME on both read paths: the default name-based path
(`hudi.parquet.use-column-names=true`) via trino-hive's `ParquetPageSourceFactory`, and the index-based
path (`hudi.parquet.use-column-names=false`) via `HudiPageSourceProvider.remapColumnIndicesToPhysical`,
which derives physical indices from a name lookup and maps a miss one past the last field so the parquet
reader null-fills it (#19288).

Name resolution covers the schema-on-write evolution Hudi performs today: an added column null-fills for
base files written before the add, which is also what the file-group reader merge expects for those
records. It does NOT cover full schema evolution written under `hoodie.schema.on.read.enable`:

- a RENAMED column no longer matches the old file's field name, so old base files read as null instead of
returning the data stored under the old name;
- TYPE changes (e.g. int -> long) are not remapped against the file's physical type.

Spark and Hive readers resolve both through the table's InternalSchema history, mapping each query column
to the file column by column id before the parquet read. The Trino connector currently reads neither the
internal-schema history nor the per-file schema commit id.

### Proposal

Port internal-schema resolution to the connector read path: when `hoodie.schema.on.read.enable` is set,
load the InternalSchema history, map requested columns to file columns by column id (rename-safe) and
apply type promotion -- on the name-based and index-based parquet paths and the file-group reader's
log-side projection alike.

### Acceptance

- Reads of a table with a renamed column return the old files' data under the new name.
- Promoted types read correctly from files written with the narrower type.
- Behavior without `hoodie.schema.on.read.enable` is unchanged.

Pointer: `hudi-trino/src/main/java/io/trino/plugin/hudi/HudiPageSourceProvider.java` (`createPageSource`,
`remapColumnIndicesToPhysical`); Spark reference: `HoodieParquetFileFormat`'s internal-schema handling.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in hudi-trino/src/main/java/io/trino/plugin/hudi/HudiPageSourceProvider.java, especially createPageSource and remapColumnIndicesToPhysical, then compare the Spark reference HoodieParquetFileFormat. Trace how the connector could use InternalSchema history and the per-file schema commit id across both parquet paths and file-group log projection. Done means rename and promoted-type reads work with schema-on-read enabled while existing behavior remains unchanged without it.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.