apache / apache/hudi

[TASK] trino: add a Spark-write / Trino-read E2E test for schema-on-write type evolution

Open
#19,743 0 comments 0 reactions 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**

Add an end-to-end integration test that produces a real Hudi type evolution with a Spark write and then reads it back through the Trino connector, so the connector's evolved-column predicate handling is covered against a table an actual write path created rather than a hand-built fixture.

Two pieces, both modelled on files that already exist:

- `docker/demo/sparksql-schema-evolution-trino.commands`, on the shape of `sparksql-blob-type-df.commands`: write a v1 schema with `metric FloatType` into `dt='2024-01-01'` with `SaveMode.Overwrite`, then append a v2 schema with `metric DoubleType` into `dt='2024-01-02'`, both through one shared `applyWriteOpts` so hive sync is configured identically, and print a `SCHEMA_EVOLUTION_TEST_SUCCESS` marker.
- `hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/trino/ITTestTrinoSchemaEvolution.java`, on the `ITTestTrinoCustomType` skeleton: `SELECT count(*)` as the anchor, `SELECT id ... WHERE metric > 1.5 ORDER BY id` returning ids from both partitions as the regression assertion, and `WHERE id > 4` as a positive control that pushdown is still alive.

No workflow change is needed: `hudi_trino_e2e.yml` already runs `-Dit.test='ITTestTrino*'` against the spark402 compose stack, which is the only pair defining the `trinocoordinator` service.

**Why this task is needed:**

#19467 fixes a query failure (#19457) where a predicate on a type-widened column either failed the split with `Corrupted statistics for column ...` or, with parquet bloom filters present, silently dropped the row groups holding the matching rows. Its tests build the post-evolution state directly: a base file written with the pre-evolution physical types, plus handles carrying the widened types the metastore reports. That is exactly the state an unrewritten base file is in, but it is asserted rather than produced.

What that leaves uncovered is the write side. Two assumptions the connector-side fix rests on are never exercised: that a DataFrame append carrying a widened schema really does leave earlier base files unrewritten, and that hive sync really does report the widened type for the whole table afterwards. Only a Spark write can establish either.

The schema-on-read route (`ALTER TABLE ... ALTER COLUMN ... TYPE` under `hoodie.schema.on.read.enable=true`) is deliberately out of scope: `hudi-trino` carries no reference to `InternalSchema` at all, so it is a separate gap, tracked in #19381.

### Task Type

Test enhancement

### Related Issues

**Parent feature issue:** #19457
**Related issues:** #19467, #19381

Contributor guide

No contributing guide indexed for this repository

Research direction

Compare docker/demo/sparksql-schema-evolution-trino.commands with sparksql-blob-type-df.commands, then read ITTestTrinoCustomType and ITTestTrinoSchemaEvolution.java. Run the existing hudi_trino_e2e.yml Trino integration target against the spark402 compose stack; done means the Spark-created Float-to-Double evolution is read successfully, the metric predicate returns ids from both partitions, and the id predicate still pushes down.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java, shell, sql
Domain
data-engineering, distributed-systems, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.