apache / apache/gluten

[Core] Spark assert_true and raise_error function support

Open
#5,991 11 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

spark `assert_true` and `raise_error` function not compatible with current velox type system and function register interface.
as below query a example,

```
val df = spark.sql("select assert_true(1 > 0) as v, id from t")
df: org.apache.spark.sql.DataFrame = [v: void, id: bigint]
```

```
df.schema
res3: org.apache.spark.sql.types.StructType = StructType(StructField(v,NullType,true),StructField(id,LongType,true))
```
it would generate `void` type and `NullType` in output schema which would convert to unknown type in velox, for `raise_error`, it expect void return type but velox function always expect a non-void return type.

One way i tried is convert `raise_error` from void return type to string type but it need we update not only expression itself but also the output type schema also from NullType to StringType, otherwise would encounter mem allocation issue based on my local test.

Update:
`RaiseError`'s data type is NullType. Ditto for `AssertTrue` as it is replaced by `IF` + `RaiseError`.
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala#L82

@zhouyuan / @PHILO-HE / @FelixYBW

Contributor guide

Open the contributing guide

Research direction

Start with Spark's sql/catalyst/expressions/misc.scala, especially RaiseError and AssertTrue, then trace Gluten's Velox function registration and output type handling. Done means both functions work with the current Velox interface without producing incompatible void or unknown schemas.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.