apache / apache/gluten

[VL] Exception when SQLConf.READ_SIDE_CHAR_PADDING config is enabled

Open
#4,294 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

A new config has been added in spark 3.4 as part of this https://github.com/apache/spark/pull/38151. While fixing TPCDS Plan Suite(https://github.com/oap-project/gluten/pull/4282) it was figured out that if this config is enabled an exception is thrown in gluten:
Unable to convert the expression staticinvoke(class org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils, StringType, readSidePadding, sm_type#106, 30, true, false, true)
java.lang.UnsupportedOperationException: Unable to convert the expression staticinvoke(class org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils, StringType, readSidePadding, sm_type#106, 30, true, false, true)
at io.substrait.spark.expression.ToSubstraitExpression.default(ToSubstraitExpression.scala:46)
at io.substrait.spark.expression.ToSubstraitExpression.translateUp(ToSubstraitExpression.scala:145)
at io.substrait.spark.expression.ToSubstraitExpression.translateUp(ToSubstraitExpression.scala:136)
at io.substrait.spark.expression.ToSubstraitExpression.convert(ToSubstraitExpression.scala:57)
at io.substrait.spark.expression.ToSubstraitExpression.apply(ToSubstraitExpression.scala:50)
at io.substrait.spark.logical.ToSubstraitRel.toExpression(ToSubstraitRel.scala:260)
at io.substrait.spark.logical.ToSubstraitRel.$anonfun$visitProject$1(ToSubstraitRel.scala:230)
at scala.collection.immutable.List.map(List.scala:297)
at io.substrait.spark.logical.ToSubstraitRel.visitProject(ToSubstraitRel.scala:230)
at io.substrait.spark.logical.ToSubstraitRel.visitProject(ToSubstraitRel.scala:45)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlanVisitor.visit(LogicalPlanVisitor.scala:38)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlanVisitor.visit$(LogicalPlanVisitor.scala:25)

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in ToSubstraitExpression.scala at the default and translateUp paths shown in the stack trace, then reproduce the failure with SQLConf.READ_SIDE_CHAR_PADDING enabled. Compare the generated expression with the TPCDS Plan Suite case that exposed it; done means the readSidePadding expression converts without an exception and the affected suite passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.