apache / apache/datafusion-comet
[Variant] Support native `schema_of_variant` and `schema_of_variant_agg`
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Spark 4.x provides `schema_of_variant` for a single value and `schema_of_variant_agg` for the merged schema of a Variant column. Comet does not implement either expression, so schema discovery over a natively scanned Variant column falls back:
```sql
SELECT schema_of_variant(v) FROM t;
SELECT schema_of_variant_agg(v) FROM t;
```
Spark's scalar implementation infers and prints the schema, using `OBJECT<...>` rather than `STRUCT<...>`, in [`SchemaOfVariant`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L778-L890). The aggregate shares that inference and merges schemas across rows and partial buffers in [`SchemaOfVariantAgg`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L892-L953).
### Describe the potential solution
Implement one shared Spark-compatible Variant schema inference/merge helper, then expose it through the scalar expression and aggregate:
- infer scalar, decimal precision/scale, date/timestamp, binary, UUID, array, object, and Variant-null (`VOID`) schemas;
- keep object fields in Spark's required order and print `OBJECT<...>` names with Spark-compatible quoting;
- merge heterogeneous array elements and object fields with Spark's compatible-type rules;
- ignore SQL NULL rows in the aggregate, start/finish an empty buffer as `VOID`, and support partial-buffer merge/serialization; and
- admit Variant only for these two expressions while preserving general fallback gates.
Add focused parity and native-plan tests for every scalar kind, JSON versus SQL NULL, nested arrays/objects, heterogeneous values, decimal widening, field-name quoting and ordering, empty/all-null inputs, grouping, and multi-partition partial aggregation.
### Additional context
Related work: #4295, #5407, #5424, and #5425.
Non-goals: schema-driven Variant casts, subfield pruning, predicate pushdown, writing, shuffle/spill of Variant values, C2R, Python transport, and Iceberg-specific work.
Contributor guide
Assessment
This issue has not been assessed yet.