apache / apache/datafusion-comet
[Variant] Support Spark's pushed one-field VariantStruct (whole-value pushVariantIntoScan rewrite) in native Parquet scans
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## What is the problem the feature request solves?
#5407 supports direct top-level `VariantType` projection through the ordinary native Parquet scan, but leaves Spark's `PushVariantIntoScan` output (`VariantStruct`) as an explicit Spark fallback.
`spark.sql.variant.pushVariantIntoScan` defaults to `false` on Spark 4.0.x but `true` (as an internal conf) on Spark 4.1.0, 4.1.3, and 4.2.0. On those versions every query that projects a Variant column is rewritten before Comet's scan rule runs, so out of the box the native Variant projection path from #5407 is unreachable: users must set the internal conf to `false` to get a native scan.
## Describe the potential solution
When the rewrite requests exactly one field with `RequestedVariantField.fullVariant` (path `$`, target type `VariantType`), the request is semantically identical to whole-value projection. Comet could:
- recognize this one shape of marked `VariantStruct` in `CometScanRule` and keep the scan native;
- emit the `struct<0: variant>` output layout Spark's rewritten expressions consume by ordinal, wrapping the normalized `[value, metadata]` Variant struct in the outer single-field struct with its per-field `VariantMetadata` preserved; and
- map that layout across the FFI boundary and the JVM vector bridge.
Multi-field, typed, and path-extracted requests remain future work under #5438.
## Additional context
Follow-up to #5407 (see the review discussion there). Related: #4295, #5438.
Contributor guide
Assessment
This issue has not been assessed yet.