apache / apache/datafusion-comet
[Variant] Support whole-value VariantType in native Parquet writes
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Comet currently falls back when an ordinary native Parquet write has a Spark `VariantType` column, even when the child already produces a whole Variant value. This prevents native scan-to-write copies such as reading `SELECT v FROM parquet_table` and writing the result back to Parquet.
Spark 4.1.3 defines the ordinary, unshredded Parquet representation as an annotated group with required binary `value` and `metadata` children ([schema conversion](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L858-L866), [value writing](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala#L282-L292)). The existing arrow-rs 59.2.0 writer already maps an Arrow Struct Field carrying the Variant extension marker to the Parquet Variant logical annotation ([extension mapping](https://github.com/apache/arrow-rs/blob/59.2.0/parquet/src/arrow/schema/extension.rs#L125-L134)).
Upstream `main` uses **DataFusion 55.0.0 and Arrow/Parquet 59.2.0** through merged [#5262](https://github.com/apache/datafusion-comet/pull/5262). The dependency upgrade is no longer a blocker. Native writer admission and the Field/storage contract remain to be implemented.
This issue is limited to whole-value, unshredded output. #3983 continues to track shredded Variant reader/writer support.
### Describe the potential solution
- Allow direct, top-level Variant fields only at the ordinary native Parquet writer boundary when the native child already provides canonical `[value, metadata]` storage.
- Preserve `ARROW:extension:name=arrow.parquet.variant`, field name/nullability, Binary child types, and child order in the writer schema.
- Reuse the existing protobuf-to-Arrow Field path and `parquet::arrow::ArrowWriter`; do not add another Variant dependency or encoder.
- Write a Parquet `VARIANT(1)` annotated group containing exactly `value` and `metadata`.
Verify Spark and Comet round trips for objects, arrays/scalars, Variant JSON null, SQL NULL, nullable parents, multiple Variant columns, and fields before/after Variant. Assert the plan contains the native writer, the Parquet footer has the Variant annotation, and Spark reads the result as `VariantType`. Preserve Spark 3.x behavior.
### Additional context
Depends on the whole-value scan/Field identity work in #5407. Variant-valued native expression producers from #5425 may later feed this writer but are not required for the scan-to-write case.
Shredded/`typed_value` output, nested Variant, Variant-producing expressions, Iceberg writes, casts, C2R, shuffle, and spill remain separate.
Contributor guide
Assessment
This issue has not been assessed yet.