Parquet: Default Avro writer fails to write Variant values
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
### Description
The default Iceberg Parquet Avro writer fails when writing a `GenericData.Record` containing a logical `Variant` value.
`Parquet.write(...).schema(schema).build()` uses `ParquetAvro.DEFAULT_MODEL`, which registers `VariantConversion`. However, Avro looks up conversions using the datum's exact runtime class. Values returned by `Variant.of(...)` implement `Variant`, but their runtime class is `VariantData`. Because `VariantConversion#getConvertedType()` returns `Variant.class`, the conversion is not found.
Avro consequently treats the value as an `IndexedRecord`, resulting in a `ClassCastException`.
### Reproduction
```java
Schema schema = new Schema(required(1, "v", Types.VariantType.get()));
GenericData.Record record =
new GenericData.Record(AvroSchemaUtil.convert(schema.asStruct(), "table"));
record.put("v", Variant.of(Variants.emptyMetadata(), Variants.of(34)));
try (FileAppender writer =
Parquet.write(Files.localOutput(file)).schema(schema).build()) {
writer.add(record);
}
```
### Actual behavior
Writing the record fails because `VariantData` cannot be cast to Avro's `IndexedRecord`.
### Expected behavior
The registered `VariantConversion` should handle any runtime class implementing `Variant`, allowing the value to be written successfully.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start by running the supplied GenericData.Record reproduction through Parquet.write(...).schema(schema).build(), then inspect ParquetAvro.DEFAULT_MODEL and VariantConversion#getConvertedType(). Done means the registered conversion handles the runtime VariantData value and the record writes successfully without the IndexedRecord ClassCastException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100