apache / apache/datafusion-comet
[Variant] Support native `to_variant_object`
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Spark 4.x exposes `to_variant_object` for converting a struct, array, or string-keyed map into a Variant object/array. Comet does not serialize or evaluate it, so construction from native columns falls back:
```sql
SELECT to_variant_object(named_struct('id', id, 'tags', tags)) FROM t;
SELECT to_variant_object(array(id, id + 1)) FROM t;
```
The SQL function is named `to_variant_object`—there is no Spark 4.1.3 function named `to_variant`. Spark registers the name in its [function registry](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L888-L896). Scalar-to-Variant conversion is expressed as `CAST(... AS VARIANT)`.
Spark restricts the root input to struct, array, or map and recursively validates supported children in [`ToVariantObject`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L121-L185). Its encoder handles supported scalar, Variant, array, string-keyed map, and struct values recursively in [`VariantExpressionEvalUtils`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala#L73-L151).
### Describe the potential solution
Add Spark-4-specific serialization and a native recursive encoder that:
- accepts exactly Spark's root and nested datatype set;
- produces canonical Variant storage with Spark-compatible object-key ordering and metadata IDs;
- preserves nested nulls as Variant null values while a null root remains SQL NULL;
- preserves embedded Variant values without losing logical identity; and
- returns the marked `[value, metadata]` output Field required by #5425.
Add focused parity and native-plan tests for structs, arrays, maps, recursively nested combinations, Variant children, SQL/nested nulls, Unicode and empty object keys, duplicate map keys, unsupported map-key types, and field ordering.
### Additional context
Related work: #4295, #5407, #5425, and the dedicated Variant-cast tracker.
Non-goals: parsing JSON strings, general casts, nested Variant column transport, writing, shuffle/spill, C2R, Python transport, and Iceberg.
Contributor guide
Assessment
This issue has not been assessed yet.