apache / apache/iceberg

Parquet: Default Avro writer fails to write Variant values

Open
#17,943 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

Spark

### Please describe the bug 🐞

### Description

The default Iceberg Parquet Avro writer fails when writing a `GenericData.Record` containing a logical `Variant` value.

`Parquet.write(...).schema(schema).build()` uses `ParquetAvro.DEFAULT_MODEL`, which registers `VariantConversion`. However, Avro looks up conversions using the datum's exact runtime class. Values returned by `Variant.of(...)` implement `Variant`, but their runtime class is `VariantData`. Because `VariantConversion#getConvertedType()` returns `Variant.class`, the conversion is not found.

Avro consequently treats the value as an `IndexedRecord`, resulting in a `ClassCastException`.

### Reproduction

```java
Schema schema = new Schema(required(1, "v", Types.VariantType.get()));

GenericData.Record record =
new GenericData.Record(AvroSchemaUtil.convert(schema.asStruct(), "table"));
record.put("v", Variant.of(Variants.emptyMetadata(), Variants.of(34)));

try (FileAppender writer =
Parquet.write(Files.localOutput(file)).schema(schema).build()) {
writer.add(record);
}
```

### Actual behavior

Writing the record fails because `VariantData` cannot be cast to Avro's `IndexedRecord`.

### Expected behavior

The registered `VariantConversion` should handle any runtime class implementing `Variant`, allowing the value to be written successfully.

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied GenericData.Record reproduction through Parquet.write(...).schema(schema).build(), then inspect ParquetAvro.DEFAULT_MODEL and VariantConversion#getConvertedType(). Done means the registered conversion handles the runtime VariantData value and the record writes successfully without the IndexedRecord ClassCastException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.