apache / apache/datafusion-comet

[Variant] Support native `parse_json` and `try_parse_json`

Open
#5,428 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Spark 4.x provides `parse_json` and `try_parse_json` to construct `VariantType` values from strings, but Comet has no serializer or native evaluator for either function. A query such as the following cannot remain native:

```sql
SELECT parse_json(json_text) FROM t;
SELECT try_parse_json(json_text) FROM t;
```

Spark implements both forms with one [`ParseJson` expression](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala#L48-L78): `parse_json` throws for malformed input while `try_parse_json` returns SQL NULL. The runtime also distinguishes malformed input and the Variant size limit and honors `spark.sql.variant.allowDuplicateKeys` in [`VariantExpressionEvalUtils.parseJson`](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala#L34-L55).

### Describe the potential solution

Add Spark-4-specific serialization and a native parser for the two expressions:

- parse each non-null input string into canonical Variant `value` and `metadata` bytes;
- emit the marked Variant output Field and Spark-compatible `[value, metadata]` children defined by #5425;
- preserve JSON null as Variant null and SQL NULL as a null parent row;
- honor Spark's duplicate-key setting, Unicode/object-key behavior, numeric representation, and Variant size limit; and
- match strict/try malformed-input behavior and Spark-compatible errors.

Add focused Spark SQL parity and plan tests for objects, arrays, scalars, JSON null, SQL NULL, Unicode keys, duplicate keys with both configuration modes, malformed JSON, oversize input, and columns before/after the result.

### Additional context

Spark registers the exact SQL names `parse_json` and `try_parse_json` in its [4.1.3 function registry](https://github.com/apache/spark/blob/v4.1.3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala#L888-L896).

Related work: #4295, #5407, and #5425.

Non-goals: `to_variant_object`, `CAST(... AS VARIANT)`, Parquet writing, nested Variant columns, shuffle/spill, C2R, Python transport, and Iceberg.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.