apache / apache/datafusion-comet
Improve performance of `in_list` expressions
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
From `CometComparisonExpressionBenchmark` (https://github.com/apache/datafusion-comet/pull/3026):
```
OpenJDK 64-Bit Server VM 17.0.17+10-Ubuntu-122.04 on Linux 6.8.0-90-generic
AMD Ryzen 9 7950X3D 16-Core Processor
in_list: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 30 32 3 35.2 28.4 1.0X
Comet (Scan) 30 32 1 35.1 28.5 1.0X
Comet (Scan + Exec) 37 38 1 28.4 35.2 0.8X
OpenJDK 64-Bit Server VM 17.0.17+10-Ubuntu-122.04 on Linux 6.8.0-90-generic
AMD Ryzen 9 7950X3D 16-Core Processor
not_in_list: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 30 32 1 34.7 28.8 1.0X
Comet (Scan) 31 33 1 34.2 29.3 1.0X
Comet (Scan + Exec) 37 38 1 28.6 34.9 0.8X
```
### Describe the potential solution
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.