apache / apache/datafusion-comet

[Variant] Support Variant-valued native expression output and two-argument variant_get

Open
#5,425 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:expressions area:ffi enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Phase A of #4295 (#5407) preserves Spark `VariantType` for a whole value coming directly from an ordinary native Parquet scan. #5424 separately tracks native `variant_get` / `try_variant_get` evaluation for literal paths and scalar, non-Variant targets.

Spark's two-argument forms default the target type to [`VariantType`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L601-L615):

```sql
SELECT variant_get(v, '$.customer') FROM t;
SELECT try_variant_get(v, '$.items[0]') FROM t;
```

These results need more than the physical Arrow `Struct` datatype. Their logical identity is the parent Field extension marker `ARROW:extension:name = arrow.parquet.variant`, which Comet's JVM bridge recognizes explicitly in [`Utils.fromArrowField`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/spark/src/main/scala/org/apache/spark/sql/comet/util/Utils.scala#L72-L96).

Comet builds native projects with DataFusion's [`ProjectionExec`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/execution/operators/projection.rs#L45-L67). DataFusion's default [`PhysicalExpr::return_field`](https://github.com/apache/datafusion/blob/55.0.0/datafusion/physical-expr-common/src/physical_expr.rs#L86-L95) constructs a Field from only the expression datatype and nullability, so a Variant-producing expression that uses the default would describe its result as an ordinary Struct. DataFusion already provides the needed propagation point: [`ProjectionExprs::project_schema`](https://github.com/apache/datafusion/blob/55.0.0/datafusion/physical-expr/src/projection.rs#L463-L488) copies metadata returned by the expression into the aliased output Field.

Comet's generic [`ScalarFunctionExpr` construction](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/execution/planner.rs#L3498-L3504) currently supplies an unannotated `Field::new(...)`, so merely adding Variant as a function return datatype would still lose the logical marker.

Without an explicit Variant result Field, a computed Variant can lose its logical identity and be imported by Spark as `StructType` instead of `VariantType`.

Upstream `main` uses **DataFusion 55.0.0 and Arrow/Parquet 59.2.0** through merged [#5262](https://github.com/apache/datafusion-comet/pull/5262). The dependency upgrade is no longer a blocker. DataFusion 55 already provides the Field propagation hook described above; the remaining work is Comet's explicit Variant expression-result contract and admission, not waiting for that upgrade.

### Describe the potential solution

Define the result contract for direct, top-level native expressions that return Variant, using existing Field-aware paths rather than adding another schema layer:

- make each admitted Variant-producing physical expression return the canonical marked Field from `return_field`, reusing Comet's [`to_arrow_field`](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/execution/serde.rs#L203-L219);
- return physical storage with exactly two ordinary Binary children in Spark order `[value, metadata]`, preserving parent nullability;
- rebuild an extracted Variant so unused metadata entries are removed, matching [Spark's Variant-target behavior](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L430-L435);
- allow `VariantType` through JVM expression/operator admission only for serializers that guarantee this contract, while retaining the general Variant fallback gates;
- rely on DataFusion's existing projection metadata propagation and #5407's [RecordBatch-Field FFI export](https://github.com/peterxcli/datafusion-comet/blob/c355fefd9c0b7e96d86523a7214bb2cdd47e1a55/native/core/src/execution/jni_api.rs#L689-L738); and
- use literal-path, two-argument `variant_get` and `try_variant_get` as the first end-to-end consumers, sharing the extraction semantics/kernel tracked by #5424.

Focused tests should verify:

- the native result's parent Field contains `arrow.parquet.variant`;
- Spark reports the output datatype as `VariantType`, not `StructType`;
- children are exactly `[value: Binary, metadata: Binary]` through aliases and with columns before/after the expression;
- object, array, scalar, Variant JSON null, SQL NULL, and nullable-parent results match Spark; and
- unsupported Variant-producing expressions and downstream consumers still fall back without losing the native scan beneath them.

### Additional context

Related work:

- #4295 — Variant scan/Iceberg umbrella
- #5407 — whole-value ordinary-Parquet scan projection and Field-preserving FFI export
- #5424 — scalar-target `variant_get` / `try_variant_get` evaluation
- #3983 — shredded storage, subfield pruning, and predicate pushdown

Non-goals: dynamic paths, nested Variant output, `parse_json` / `to_variant` and other Variant producers, C2R, shuffle/spill, Python transport, writes, and Iceberg. Supporting one direct Variant-valued native projection must not enable Variant indiscriminately for other operators.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.