typelevel / typelevel/frameless
UDF fails when subexpression elimination is used in interpreted mode
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 895
- Forks
- 135
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 3
Description
As UDF expects compilation always this fails when a child cannot compile, such as ExpressionProxy.
If, in interpreted mode, a child is deemed an identical subexpression (as occurs in the UdfTests."multiple two argument udf" test) spark is able to swap the contents as an optimisation to reduce re-calculation.
In the case where the analyser deems compilation the correct approach this isn't an issue. Targeted fix as part of #800.
Known to occur on Databricks 11.3 and other lower runtimes, fixed as part of 3.4 in SPARK-41991 .
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the UdfTests."multiple two argument udf" case and trace how interpreted mode handles an identical child subexpression that cannot compile, such as ExpressionProxy. Review the targeted fix mentioned in #800 and the related Spark 3.4 change in SPARK-41991. Done means the interpreted UDF path no longer fails and the regression test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100