drop-spark4.0: apply variantType() in BaseSpark4Adapter and remove parquet 1.15 variant shims
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Background
Spark 4.0 ships parquet 1.15.2, which does not have the variant `LogicalTypeAnnotation` (added in parquet 1.16.0+). Spark 4.1 ships parquet 1.16+.
To keep variant writes working on both, `BaseSpark4Adapter.applyVariantLogicalType` is a no-op default and `Spark4_1Adapter` overrides it to actually apply `variantType()`. There is also a Spark 4.0-only read-support shim (`Spark40HoodieParquetReadSupport`) kept for the same reason.
### Cleanup to do once Spark 4.0 support is dropped
When all remaining 4.x adapters are on parquet 1.16+:
- Apply `variantType()` directly in `BaseSpark4Adapter.applyVariantLogicalType` and delete the no-op default plus the `Spark4_1Adapter` override. See `BaseSpark4Adapter.scala`.
- Remove the `hudi-spark4.0.x` module shims, e.g. `Spark40HoodieParquetReadSupport` (`Spark40HoodieParquetReadSupport.scala`).
### Related
- PR #18065 (introduced the no-op default / override split for variant shredding)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.