apache / apache/datafusion-comet
[Variant] Support Variant-valued native expression output and two-argument variant_get
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Phase A of #4295 (#5407) preserves Spark `VariantType` for a whole value coming directly from an ordinary native Parquet scan. #5424 separately tracks native `variant_get` / `try_variant_get` evaluation for literal paths and scalar, non-Variant targets.
Spark's two-argument forms default the target type to [`VariantType`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L601-L615):
```sql
SELECT variant_get(v, '$.customer') FROM t;
SELECT try_variant_get(v, '$.items[0]') FROM t;
```
These results need more than the physical Arrow `Struct` datatype. Their logical identity is the parent Field extension marker `ARROW:extension:name = arrow.parquet.variant`, which Comet's JVM bridge recognizes explicitly in [`Utils.fromArrowField`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala#L72-L96).
Comet builds native projects with DataFusion's [`ProjectionExec`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/execution/operators/projection.rs#L45-L67). DataFusion's default [`PhysicalExpr::return_field`](https://github.com/apache/datafusion/blob/55.0.0/datafusion/physical-expr-common/src/physical_expr.rs#L86-L95) constructs a Field from only the expression datatype and nullability, so a Variant-producing expression that uses the default would describe its result as an ordinary Struct. DataFusion already provides the needed propagation point: [`ProjectionExprs::project_schema`](https://github.com/apache/datafusion/blob/55.0.0/datafusion/physical-expr/src/projection.rs#L463-L488) copies metadata returned by the expression into the aliased output Field.
Comet's generic [`ScalarFunctionExpr` construction](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/execution/planner.rs#L3498-L3504) currently supplies an unannotated `Field::new(...)`, so merely adding Variant as a function return datatype would still lose the logical marker.
Without an explicit Variant result Field, a computed Variant can lose its logical identity and be imported by Spark as `StructType` instead of `VariantType`.
Upstream `main` uses **DataFusion 55.0.0 and Arrow/Parquet 59.2.0** through merged [#5262](https://github.com/apache/datafusion-comet/pull/5262). The dependency upgrade is no longer a blocker. DataFusion 55 already provides the Field propagation hook described above; the remaining work is Comet's explicit Variant expression-result contract and admission, not waiting for that upgrade.
### Describe the potential solution
Define the result contract for direct, top-level native expressions that return Variant, using existing Field-aware paths rather than adding another schema layer:
- make each admitted Variant-producing physical expression return the canonical marked Field from `return_field`, reusing Comet's [`to_arrow_field`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/execution/serde.rs#L203-L219);
- return physical storage with exactly two ordinary Binary children in Spark order `[value, metadata]`, preserving parent nullability;
- rebuild an extracted Variant so unused metadata entries are removed, matching [Spark's Variant-target behavior](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L430-L435);
- allow `VariantType` through JVM expression/operator admission only for serializers that guarantee this contract, while retaining the general Variant fallback gates;
- rely on DataFusion's existing projection metadata propagation and #5407's [RecordBatch-Field FFI export](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/execution/jni_api.rs#L689-L738); and
- use literal-path, two-argument `variant_get` and `try_variant_get` as the first end-to-end consumers, sharing the extraction semantics/kernel tracked by #5424.
Focused tests should verify:
- the native result's parent Field contains `arrow.parquet.variant`;
- Spark reports the output datatype as `VariantType`, not `StructType`;
- children are exactly `[value: Binary, metadata: Binary]` through aliases and with columns before/after the expression;
- object, array, scalar, Variant JSON null, SQL NULL, and nullable-parent results match Spark; and
- unsupported Variant-producing expressions and downstream consumers still fall back without losing the native scan beneath them.
### Additional context
Related work:
- #4295 — Variant scan/Iceberg umbrella
- #5407 — whole-value ordinary-Parquet scan projection and Field-preserving FFI export
- #5424 — scalar-target `variant_get` / `try_variant_get` evaluation
- #3983 — shredded storage, subfield pruning, and predicate pushdown
Non-goals: dynamic paths, nested Variant output, `parse_json` / `to_variant` and other Variant producers, C2R, shuffle/spill, Python transport, writes, and Iceberg. Supporting one direct Variant-valued native projection must not enable Variant indiscriminately for other operators.
Contributor guide
Assessment
This issue has not been assessed yet.