Spark 4 variant shredding: FIXED typed_value broken on both write and read paths
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Describe the problem
`Spark4VariantShreddingProvider` handles Avro FIXED `typed_value` leaves incorrectly in both directions; the behavior is latent since #18065 (`23fe7bfcd7e6`).
- **Write**: `avroTypeToScalarType` maps FIXED to `BinaryType`, and `convertScalarToAvro` then returns a `ByteBuffer`. That record fails `ConvertingGenericData.INSTANCE.validate(...)` (the data model `HoodieAvroWriteSupport` hands to parquet-avro), and writing throws `ClassCastException: HeapByteBuffer cannot be cast to GenericFixed`.
- **Read**: a FIXED column decodes to `GenericData$Fixed`, which the `(ByteBuffer)` cast in `AvroVariantRow.getBinary` rejects with a `ClassCastException`.
- **No size check**: `avroTypeToScalarType` ignores `getFixedSize()`, so a payload of the wrong length shreds without complaint.
- Same hole class for the spec-canonical uuid-on-`fixed(16)` encoding: shred writes a `String` (`UnresolvedUnionException` at write), and rebuild on `GenericData$Fixed` fails with `IllegalArgumentException: UUID string too large`.
### Reachability
Not reachable from the shredding DDL (`HoodieAvroWriteSupport.parseSimpleType` only accepts int/long/string/double/float/boolean/binary/decimal(p,s)), but reachable when the table Avro schema declares a FIXED `typed_value` directly (the shredding-enabled, no-forced-schema branch of `generateEffectiveSchema` uses the schema as-is), and on the read path for files written by other engines.
### Suggested fix
Either:
1. Return null for FIXED in `avroTypeToScalarType` (decline to shred; simplest and safe), or
2. Emit `GenericData.Fixed` in `convertScalarToAvro`, accept `GenericFixed` in `getBinary`/`getUuid`, and enforce `getFixedSize()`.
The unit test pinning the broken behavior was removed in PR #19403 review; whichever fix lands should add a test asserting the chosen behavior (decline-to-residual or a real `GenericData.Fixed` round trip) plus `ConvertingGenericData.INSTANCE.validate` on the result.
### Related cleanups
- The `Byte`/`Short`/`Integer` widening arms in `convertScalarToAvro` and `AvroVariantRow.getByte`/`getShort` are unreachable (Avro has no byte/short; `avroTypeToScalarType` only emits `IntegralSize.INT`/`LONG`) - delete them.
- No test writes a shredded record through `HoodieAvroWriteSupport` to parquet and reads it back through `HoodieAvroParquetReader`; add functional coverage for the DDL-expressible `binary` and `decimal(10,2)` leaves so serde-boundary regressions cannot ship silently.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.