apache / apache/datafusion-comet
[Variant] Support native Parquet projection of VariantType nested in struct, array, and map columns
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
#5407 adds direct, top-level whole-value Variant projection for ordinary native Parquet scans. Spark also permits Variant nested inside structs, arrays, and string-keyed maps, and its Parquet Variant suite covers `struct<... variant>`, `array`, and `map` shapes ([Spark tests](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVariantShreddingSuite.scala#L200-L223)).
Comet's current scan admission only special-cases a directly top-level Variant field ([CometScanRule](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala#L963-L985)), and native normalization runs only when the target Field itself is marked Variant ([cast_column.rs](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/parquet/cast_column.rs#L545-L553)). Nested Variant therefore remains a full scan fallback.
### Describe the potential solution
- Recursively identify explicitly marked Variant leaf Fields while adapting ordinary Parquet batches.
- Apply the existing whole-value dictionary decode, unshredding, Binary normalization, Spark object ordering, and `[value, metadata]` reordering exactly once at each Variant leaf.
- Preserve every enclosing struct field, list/map offset, container and element null bitmap, field name/nullability, and extension marker.
- Admit only Spark-supported logical shapes; keep Variant map keys and malformed/unmarked lookalike Structs rejected.
Add Spark parity and vector-layout tests for nullable struct parents, array elements, and map values; shredded and unshredded input; SQL NULL and Variant JSON null; and fields before/after each nested Variant. Verify pruning an unread nested Variant remains compatible with #5377.
### Additional context
#3983 continues to own broader shredded writer, subfield-pruning, and predicate-pushdown work. This issue covers full-value ordinary-Parquet projection only.
Nested Variant expressions, C2R, shuffle/spill, Python, writes, and Iceberg remain separate.
Contributor guide
Assessment
This issue has not been assessed yet.