apache / apache/datafusion-comet

[Feature] Support Spark expression: json_tuple

Open
#3,160 1 comment 0 reactions 1 assignee Claimed by @CuteChuanChuan View on GitHub
enhancement json expressions
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## What is the problem the feature request solves?

> **Note:** This issue was generated with AI assistance. The specification details have been extracted from Spark documentation and may need verification.

Comet does not currently support the Spark `json_tuple` function, causing queries using this function to fall back to Spark's JVM execution instead of running natively on DataFusion.

JsonTuple is a generator expression that extracts multiple fields from a JSON string and returns them as separate columns in a tuple format. It takes a JSON string as the first argument followed by one or more field path expressions, returning a row with the extracted values in the order specified.

Supporting this expression would allow more Spark workloads to benefit from Comet's native acceleration.

## Describe the potential solution

### Spark Specification

**Syntax:**
```sql
json_tuple(json_string, field1, field2, ...)
```

**Arguments:**
| Argument | Type | Description |
|----------|------|-------------|
| json_string | String | The JSON string to parse and extract fields from |
| field1, field2, ... | String | One or more field names/paths to extract from the JSON |

**Return Type:** Returns a StructType with columns named `c0`, `c1`, `c2`, etc., where each column corresponds to the extracted field values. All output columns have nullable string type matching the input JSON expression's data type.

**Supported Data Types:**
Input arguments must be string types with collation support (specifically StringTypeWithCollation with trimming support). All children expressions must accept string data types.

**Edge Cases:**
- Always returns at least one row (nullable = false), even for invalid JSON
- Null JSON input results in null values for all extracted fields
- Invalid or missing field paths return null for those specific columns
- Foldable (constant) field expressions are pre-evaluated for performance optimization
- Requires minimum 2 arguments (JSON string + at least one field name), throws QueryCompilationErrors.wrongNumArgsError otherwise

**Examples:**
```sql
-- Extract name and age from JSON
SELECT json_tuple('{"name":"John", "age":30, "city":"NYC"}', 'name', 'age') AS (name, age);
-- Returns: John, 30

-- Extract nested fields
SELECT json_tuple('{"user":{"id":123,"email":"test@example.com"}}', 'user.id', 'user.email') AS (id, email);
```

```scala
// DataFrame API usage
import org.apache.spark.sql.functions._
df.select(json_tuple(col("json_col"), lit("field1"), lit("field2")).as(Seq("c0", "c1")))
```

### Implementation Approach

See the [Comet guide on adding new expressions](https://datafusion.apache.org/comet/contributor-guide/adding_a_new_expression.html) for detailed instructions.

1. **Scala Serde**: Add expression handler in `spark/src/main/scala/org/apache/comet/serde/`
2. **Register**: Add to appropriate map in `QueryPlanSerde.scala`
3. **Protobuf**: Add message type in `native/proto/src/proto/expr.proto` if needed
4. **Rust**: Implement in `native/spark-expr/src/` (check if DataFusion has built-in support first)

## Additional context

**Difficulty:** Large
**Spark Expression Class:** `org.apache.spark.sql.catalyst.expressions.JsonTuple`

**Related:**
- `get_json_object` - Extract single JSON field
- `from_json` - Parse JSON with explicit schema
- `json_extract` - Alternative JSON field extraction

---
*This issue was auto-generated from Spark reference documentation.*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.