[TASK] trino: add a Spark-write / Trino-read E2E test for schema-on-write type evolution
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Task Description
**What needs to be done:**
Add an end-to-end integration test that produces a real Hudi type evolution with a Spark write and then reads it back through the Trino connector, so the connector's evolved-column predicate handling is covered against a table an actual write path created rather than a hand-built fixture.
Two pieces, both modelled on files that already exist:
- `docker/demo/sparksql-schema-evolution-trino.commands`, on the shape of `sparksql-blob-type-df.commands`: write a v1 schema with `metric FloatType` into `dt='2024-01-01'` with `SaveMode.Overwrite`, then append a v2 schema with `metric DoubleType` into `dt='2024-01-02'`, both through one shared `applyWriteOpts` so hive sync is configured identically, and print a `SCHEMA_EVOLUTION_TEST_SUCCESS` marker.
- `hudi-integ-test/src/test/java/org/apache/hudi/integ2/testcontainers/trino/ITTestTrinoSchemaEvolution.java`, on the `ITTestTrinoCustomType` skeleton: `SELECT count(*)` as the anchor, `SELECT id ... WHERE metric > 1.5 ORDER BY id` returning ids from both partitions as the regression assertion, and `WHERE id > 4` as a positive control that pushdown is still alive.
No workflow change is needed: `hudi_trino_e2e.yml` already runs `-Dit.test='ITTestTrino*'` against the spark402 compose stack, which is the only pair defining the `trinocoordinator` service.
**Why this task is needed:**
#19467 fixes a query failure (#19457) where a predicate on a type-widened column either failed the split with `Corrupted statistics for column ...` or, with parquet bloom filters present, silently dropped the row groups holding the matching rows. Its tests build the post-evolution state directly: a base file written with the pre-evolution physical types, plus handles carrying the widened types the metastore reports. That is exactly the state an unrewritten base file is in, but it is asserted rather than produced.
What that leaves uncovered is the write side. Two assumptions the connector-side fix rests on are never exercised: that a DataFrame append carrying a widened schema really does leave earlier base files unrewritten, and that hive sync really does report the widened type for the whole table afterwards. Only a Spark write can establish either.
The schema-on-read route (`ALTER TABLE ... ALTER COLUMN ... TYPE` under `hoodie.schema.on.read.enable=true`) is deliberately out of scope: `hudi-trino` carries no reference to `InternalSchema` at all, so it is a separate gap, tracked in #19381.
### Task Type
Test enhancement
### Related Issues
**Parent feature issue:** #19457
**Related issues:** #19467, #19381
Contributor guide
No contributing guide indexed for this repository
Research direction
Compare docker/demo/sparksql-schema-evolution-trino.commands with sparksql-blob-type-df.commands, then read ITTestTrinoCustomType and ITTestTrinoSchemaEvolution.java. Run the existing hudi_trino_e2e.yml Trino integration target against the spark402 compose stack; done means the Spark-created Float-to-Double evolution is read successfully, the metric predicate returns ids from both partitions, and the id predicate still pushes down.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java, shell, sql
- Domain
- data-engineering, distributed-systems, testing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100