apache / apache/hudi

drop-spark4.0: apply variantType() in BaseSpark4Adapter and remove parquet 1.15 variant shims

Open
#18,935 1 comment 0 reactions 1 assignee Claimed by @voonhous View on GitHub
area:schema type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Background

Spark 4.0 ships parquet 1.15.2, which does not have the variant `LogicalTypeAnnotation` (added in parquet 1.16.0+). Spark 4.1 ships parquet 1.16+.

To keep variant writes working on both, `BaseSpark4Adapter.applyVariantLogicalType` is a no-op default and `Spark4_1Adapter` overrides it to actually apply `variantType()`. There is also a Spark 4.0-only read-support shim (`Spark40HoodieParquetReadSupport`) kept for the same reason.

### Cleanup to do once Spark 4.0 support is dropped

When all remaining 4.x adapters are on parquet 1.16+:

- Apply `variantType()` directly in `BaseSpark4Adapter.applyVariantLogicalType` and delete the no-op default plus the `Spark4_1Adapter` override. See `BaseSpark4Adapter.scala`.
- Remove the `hudi-spark4.0.x` module shims, e.g. `Spark40HoodieParquetReadSupport` (`Spark40HoodieParquetReadSupport.scala`).

### Related

- PR #18065 (introduced the no-op default / override split for variant shredding)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.