apache / apache/datafusion-comet

[Variant] Support native `to_variant_object`

Open
#5,431 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Spark 4.x exposes `to_variant_object` for converting a struct, array, or string-keyed map into a Variant object/array. Comet does not serialize or evaluate it, so construction from native columns falls back:

```sql
SELECT to_variant_object(named_struct('id', id, 'tags', tags)) FROM t;
SELECT to_variant_object(array(id, id + 1)) FROM t;
```

The SQL function is named `to_variant_object`—there is no Spark 4.1.3 function named `to_variant`. Spark registers the name in its [function registry](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L888-L896). Scalar-to-Variant conversion is expressed as `CAST(... AS VARIANT)`.

Spark restricts the root input to struct, array, or map and recursively validates supported children in [`ToVariantObject`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L121-L185). Its encoder handles supported scalar, Variant, array, string-keyed map, and struct values recursively in [`VariantExpressionEvalUtils`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala#L73-L151).

### Describe the potential solution

Add Spark-4-specific serialization and a native recursive encoder that:

- accepts exactly Spark's root and nested datatype set;
- produces canonical Variant storage with Spark-compatible object-key ordering and metadata IDs;
- preserves nested nulls as Variant null values while a null root remains SQL NULL;
- preserves embedded Variant values without losing logical identity; and
- returns the marked `[value, metadata]` output Field required by #5425.

Add focused parity and native-plan tests for structs, arrays, maps, recursively nested combinations, Variant children, SQL/nested nulls, Unicode and empty object keys, duplicate map keys, unsupported map-key types, and field ordering.

### Additional context

Related work: #4295, #5407, #5425, and the dedicated Variant-cast tracker.

Non-goals: parsing JSON strings, general casts, nested Variant column transport, writing, shuffle/spill, C2R, Python transport, and Iceberg.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.