apache / apache/datafusion-comet

Improve performance of `sha` hashing expressions

Open
#3,029 3 comments 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

```
OpenJDK 64-Bit Server VM 17.0.17+10-Ubuntu-122.04 on Linux 6.8.0-90-generic
AMD Ryzen 9 7950X3D 16-Core Processor
sha1: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 183 185 2 5.7 174.2 1.0X
Comet (Scan) 178 180 2 5.9 169.7 1.0X
Comet (Scan + Exec) 340 351 17 3.1 324.3 0.5X

OpenJDK 64-Bit Server VM 17.0.17+10-Ubuntu-122.04 on Linux 6.8.0-90-generic
AMD Ryzen 9 7950X3D 16-Core Processor
sha2_224: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 189 193 2 5.6 180.1 1.0X
Comet (Scan) 181 185 2 5.8 172.2 1.0X
Comet (Scan + Exec) 435 437 1 2.4 415.1 0.4X

OpenJDK 64-Bit Server VM 17.0.17+10-Ubuntu-122.04 on Linux 6.8.0-90-generic
AMD Ryzen 9 7950X3D 16-Core Processor
sha2_256: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 198 207 13 5.3 188.6 1.0X
Comet (Scan) 186 201 18 5.6 177.8 1.1X
Comet (Scan + Exec) 481 483 2 2.2 458.6 0.4X

OpenJDK 64-Bit Server VM 17.0.17+10-Ubuntu-122.04 on Linux 6.8.0-90-generic
AMD Ryzen 9 7950X3D 16-Core Processor
sha2_384: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 311 316 3 3.4 296.8 1.0X
Comet (Scan) 313 317 3 3.3 298.7 1.0X
Comet (Scan + Exec) 778 779 1 1.3 742.3 0.4X

OpenJDK 64-Bit Server VM 17.0.17+10-Ubuntu-122.04 on Linux 6.8.0-90-generic
AMD Ryzen 9 7950X3D 16-Core Processor
sha2_512: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
Spark 322 323 2 3.3 306.9 1.0X
Comet (Scan) 316 318 1 3.3 301.3 1.0X
Comet (Scan + Exec) 958 984 26 1.1 913.7 0.3X
```

### Describe the potential solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.