apache / apache/datafusion-comet

[Variant] Support Spark's pushed one-field VariantStruct (whole-value pushVariantIntoScan rewrite) in native Parquet scans

Open
#5,519 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:scan enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## What is the problem the feature request solves?

#5407 supports direct top-level `VariantType` projection through the ordinary native Parquet scan, but leaves Spark's `PushVariantIntoScan` output (`VariantStruct`) as an explicit Spark fallback.

`spark.sql.variant.pushVariantIntoScan` defaults to `false` on Spark 4.0.x but `true` (as an internal conf) on Spark 4.1.0, 4.1.3, and 4.2.0. On those versions every query that projects a Variant column is rewritten before Comet's scan rule runs, so out of the box the native Variant projection path from #5407 is unreachable: users must set the internal conf to `false` to get a native scan.

## Describe the potential solution

When the rewrite requests exactly one field with `RequestedVariantField.fullVariant` (path `$`, target type `VariantType`), the request is semantically identical to whole-value projection. Comet could:

- recognize this one shape of marked `VariantStruct` in `CometScanRule` and keep the scan native;
- emit the `struct<0: variant>` output layout Spark's rewritten expressions consume by ordinal, wrapping the normalized `[value, metadata]` Variant struct in the outer single-field struct with its per-field `VariantMetadata` preserved; and
- map that layout across the FFI boundary and the JVM vector bridge.

Multi-field, typed, and path-extracted requests remain future work under #5438.

## Additional context

Follow-up to #5407 (see the review discussion there). Related: #4295, #5438.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.