apache / apache/datafusion-comet
Support higher-order array functions via JVM UDF bridge
Open
area:expressions
area:udf
priority:medium
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Higher-order functions (`exists`, `forall`, `transform`, `filter`, `aggregate`) fall back to Spark because they involve lambda expressions. This causes C2R/R2C transitions even when the rest of the pipeline is native.
### Describe the potential solution
Use the JVM UDF bridge (#4170) to evaluate lambdas on the JVM while keeping data in columnar Arrow format. Prototype for `exists` in #4223.
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.