apache / apache/datafusion-comet
[Variant] Complete native `variant_get` and `try_variant_get` support for dynamic paths and nested targets
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
#5424 tracks native `variant_get` / `try_variant_get` for literal paths and scalar, non-Variant targets, while #5425 tracks the two-argument Variant-returning form. Spark also supports a path supplied by a column and nested target types:
```sql
SELECT variant_get(v, path, 'bigint') FROM t;
SELECT try_variant_get(v, '$.items', 'array') FROM t;
SELECT variant_get(v, '$.customer', 'struct') FROM t;
```
Spark 4.1.3 evaluates a foldable path once but has a separate per-row path branch for non-foldable expressions in [`VariantGet`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L263-L325) and its [UTF8 path helper](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L391-L403). It accepts arrays, maps with string keys, and structs recursively as target types in [`checkDataType`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L345-L362), with the corresponding recursive casts implemented for [arrays, maps, and structs](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L495-L535).
Without these paths, otherwise supported Variant extraction queries still cross back to Spark.
### Describe the potential solution
Extend the expression-specific implementation from #5424/#5425 without enabling Variant in Comet's general expression gates:
- evaluate and parse a non-foldable string path per non-null row, preserving Spark's invalid-path error and SQL NULL behavior;
- support exactly Spark's recursive target-type set: arrays, maps with string keys, and structs whose children are supported;
- reproduce Spark's missing-field, extra-field, nested null, JSON null, decimal, datetime, and time-zone behavior;
- preserve the strict/try distinction recursively: `variant_get` raises on conversion failure, while `try_variant_get` returns SQL NULL at the failing value; and
- keep unsupported target types and shapes on explicit Spark fallback.
Add focused Spark SQL parity and plan tests for column paths, mixed paths across rows, invalid and SQL NULL paths, nested arrays/maps/structs, missing object fields, conversion failures, and nullable nested values.
### Additional context
Related work: #4295, #3983, #5407, #5424, and #5425.
Non-goals: predicate pushdown or subfield pruning, Variant constructors, general casts, generators, shuffle/spill, writing, Python transport, and Iceberg-specific work.
Contributor guide
Assessment
This issue has not been assessed yet.