[FEATURE] hudi-trino: resolve parquet columns through the internal schema for full schema evolution (hoodie.schema.on.read.enable)
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Sub-task of #18780 (RFC-105 Trino connector migration).
### Context
The connector resolves parquet columns by NAME on both read paths: the default name-based path
(`hudi.parquet.use-column-names=true`) via trino-hive's `ParquetPageSourceFactory`, and the index-based
path (`hudi.parquet.use-column-names=false`) via `HudiPageSourceProvider.remapColumnIndicesToPhysical`,
which derives physical indices from a name lookup and maps a miss one past the last field so the parquet
reader null-fills it (#19288).
Name resolution covers the schema-on-write evolution Hudi performs today: an added column null-fills for
base files written before the add, which is also what the file-group reader merge expects for those
records. It does NOT cover full schema evolution written under `hoodie.schema.on.read.enable`:
- a RENAMED column no longer matches the old file's field name, so old base files read as null instead of
returning the data stored under the old name;
- TYPE changes (e.g. int -> long) are not remapped against the file's physical type.
Spark and Hive readers resolve both through the table's InternalSchema history, mapping each query column
to the file column by column id before the parquet read. The Trino connector currently reads neither the
internal-schema history nor the per-file schema commit id.
### Proposal
Port internal-schema resolution to the connector read path: when `hoodie.schema.on.read.enable` is set,
load the InternalSchema history, map requested columns to file columns by column id (rename-safe) and
apply type promotion -- on the name-based and index-based parquet paths and the file-group reader's
log-side projection alike.
### Acceptance
- Reads of a table with a renamed column return the old files' data under the new name.
- Promoted types read correctly from files written with the narrower type.
- Behavior without `hoodie.schema.on.read.enable` is unchanged.
Pointer: `hudi-trino/src/main/java/io/trino/plugin/hudi/HudiPageSourceProvider.java` (`createPageSource`,
`remapColumnIndicesToPhysical`); Spark reference: `HoodieParquetFileFormat`'s internal-schema handling.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in hudi-trino/src/main/java/io/trino/plugin/hudi/HudiPageSourceProvider.java, especially createPageSource and remapColumnIndicesToPhysical, then compare the Spark reference HoodieParquetFileFormat. Trace how the connector could use InternalSchema history and the per-file schema commit id across both parquet paths and file-group log projection. Done means rename and promoted-type reads work with schema-on-read enabled while existing behavior remains unchanged without it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100