apache / apache/datafusion-comet

[Variant] Support native `schema_of_variant` and `schema_of_variant_agg`

Open
#5,427 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:aggregation area:expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Spark 4.x provides `schema_of_variant` for a single value and `schema_of_variant_agg` for the merged schema of a Variant column. Comet does not implement either expression, so schema discovery over a natively scanned Variant column falls back:

```sql
SELECT schema_of_variant(v) FROM t;
SELECT schema_of_variant_agg(v) FROM t;
```

Spark's scalar implementation infers and prints the schema, using `OBJECT<...>` rather than `STRUCT<...>`, in [`SchemaOfVariant`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L778-L890). The aggregate shares that inference and merges schemas across rows and partial buffers in [`SchemaOfVariantAgg`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L892-L953).

### Describe the potential solution

Implement one shared Spark-compatible Variant schema inference/merge helper, then expose it through the scalar expression and aggregate:

- infer scalar, decimal precision/scale, date/timestamp, binary, UUID, array, object, and Variant-null (`VOID`) schemas;
- keep object fields in Spark's required order and print `OBJECT<...>` names with Spark-compatible quoting;
- merge heterogeneous array elements and object fields with Spark's compatible-type rules;
- ignore SQL NULL rows in the aggregate, start/finish an empty buffer as `VOID`, and support partial-buffer merge/serialization; and
- admit Variant only for these two expressions while preserving general fallback gates.

Add focused parity and native-plan tests for every scalar kind, JSON versus SQL NULL, nested arrays/objects, heterogeneous values, decimal widening, field-name quoting and ordering, empty/all-null inputs, grouping, and multi-partition partial aggregation.

### Additional context

Related work: #4295, #5407, #5424, and #5425.

Non-goals: schema-driven Variant casts, subfield pruning, predicate pushdown, writing, shuffle/spill of Variant values, C2R, Python transport, and Iceberg-specific work.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.