apache / apache/datafusion-comet

CometProject and CometHashAggregate do not perform cross-sibling subexpression elimination over ScalaUDF

Open
#4,516 0 comments 0 reactions 0 assignees View on GitHub
area:aggregation area:expressions priority:medium
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

When a `ScalaUDF` is dispatched into the native plan via the JVM Scala UDF codegen dispatcher (enabled by default in #4514), Comet's `CometProject` and `CometHashAggregate` do not implement Spark's cross-sibling common subexpression elimination over `ScalaUDF`. An expression such as `sum(udf(b) + udf(b) + udf(b))` therefore invokes the UDF body once per reference instead of once.

This is observable in Spark's `SQLQuerySuite` "Common subexpression elimination" test: the call count for the aggregate case is 3 under Comet versus 1 in vanilla Spark. The query result is unchanged; only the number of UDF invocations differs.

Follow-on from #4514. We should extend the cross-sibling CSE that `CometProject` performs to the aggregate operator's input projection (and any other operator that builds an input projection) so that repeated `ScalaUDF` references are evaluated once.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.