[VL] Exception when SQLConf.READ_SIDE_CHAR_PADDING config is enabled
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
A new config has been added in spark 3.4 as part of this https://github.com/apache/spark/pull/38151. While fixing TPCDS Plan Suite(https://github.com/oap-project/gluten/pull/4282) it was figured out that if this config is enabled an exception is thrown in gluten:
Unable to convert the expression staticinvoke(class org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils, StringType, readSidePadding, sm_type#106, 30, true, false, true)
java.lang.UnsupportedOperationException: Unable to convert the expression staticinvoke(class org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils, StringType, readSidePadding, sm_type#106, 30, true, false, true)
at io.substrait.spark.expression.ToSubstraitExpression.default(ToSubstraitExpression.scala:46)
at io.substrait.spark.expression.ToSubstraitExpression.translateUp(ToSubstraitExpression.scala:145)
at io.substrait.spark.expression.ToSubstraitExpression.translateUp(ToSubstraitExpression.scala:136)
at io.substrait.spark.expression.ToSubstraitExpression.convert(ToSubstraitExpression.scala:57)
at io.substrait.spark.expression.ToSubstraitExpression.apply(ToSubstraitExpression.scala:50)
at io.substrait.spark.logical.ToSubstraitRel.toExpression(ToSubstraitRel.scala:260)
at io.substrait.spark.logical.ToSubstraitRel.$anonfun$visitProject$1(ToSubstraitRel.scala:230)
at scala.collection.immutable.List.map(List.scala:297)
at io.substrait.spark.logical.ToSubstraitRel.visitProject(ToSubstraitRel.scala:230)
at io.substrait.spark.logical.ToSubstraitRel.visitProject(ToSubstraitRel.scala:45)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlanVisitor.visit(LogicalPlanVisitor.scala:38)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlanVisitor.visit$(LogicalPlanVisitor.scala:25)
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
_No response_
Contributor guide
Research direction
Start in ToSubstraitExpression.scala at the default and translateUp paths shown in the stack trace, then reproduce the failure with SQLConf.READ_SIDE_CHAR_PADDING enabled. Compare the generated expression with the TPCDS Plan Suite case that exposed it; done means the readSidePadding expression converts without an exception and the affected suite passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100