ashvardanian / ashvardanian/HashEvals
Getting extremely different benchmarks from the README
- Dominant language
- Rust
- Stars
- 10
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
I ran 1 million samples as instructed in the README and this was what I got. More importantly, it shows that `StringZilla` is way slower than shown in the README. 10994.1 MiB/s (README) vs 3784.7 MiB/s (actuality)
```bash
Hash Quality Analysis:
Function | Avg.Bias | Worst.Bias | Integral ⨳ | Chi² | Throughput
-------------+------------+------------+------------+----------+---------------
Blake3 | 0.16303 % | 3.75977 % | 41.194 % | 1828.047 | 586.3 MiB/s
SeaHash | 0.18943 % | 4.44336 % | 40.876 % | 1806.709 | 3578.1 MiB/s
aHash | 0.19246 % | 4.54102 % | 40.711 % | 1771.800 | 9364.9 MiB/s
SipHash | 0.20437 % | 4.88281 % | 40.531 % | 1806.103 | 2869.7 MiB/s
FoldHash | 0.20813 % | 4.88281 % | 41.223 % | 1788.240 | 10334.5 MiB/s
FarmHash | 0.21102 % | 5.07812 % | 40.973 % | 1778.461 | 5306.1 MiB/s
xxHash3 | 0.21338 % | 5.07812 % | 40.927 % | 1826.013 | 8549.2 MiB/s
gxHash | 0.22482 % | 5.41992 % | 40.801 % | 1792.170 | 9798.4 MiB/s
StringZilla | 0.22534 % | 5.51758 % | 40.779 % | 1801.158 | 3784.7 MiB/s
MurMur3 | 0.23100 % | 5.56641 % | 41.044 % | 1790.971 | 3905.9 MiB/s
FxHash | 1.86875 % | 6.92558 % | 40.986 % | 1814.495 | 9140.8 MiB/s
Crc32 | 15.87577 % | 37.50000 % | 41.784 % | 1299.847 | 2292.7 MiB/s
RabinKarp32 | 50.00000 % | 50.00000 % | 38.362 % | 1303.198 | 338.5 MiB/s
Hash Quality Analysis - Tiny N-grams (≤ 8 bytes):
Function | Avg.Bias | Worst.Bias | Integral ⨳ | Chi² | Throughput
-------------+------------+------------+------------+----------+--------------
Blake3 | 0.51103 % | 3.75977 % | 41.194 % | 3725.665 | 69.7 MiB/s
SeaHash | 0.60072 % | 4.44336 % | 40.876 % | 3639.012 | 600.1 MiB/s
aHash | 0.61103 % | 4.54102 % | 40.711 % | 3570.448 | 1096.3 MiB/s
SipHash | 0.65208 % | 4.88281 % | 40.531 % | 3650.411 | 541.2 MiB/s
FoldHash | 0.66464 % | 4.88281 % | 41.223 % | 3636.436 | 1175.4 MiB/s
FarmHash | 0.67278 % | 5.07812 % | 40.973 % | 3597.411 | 765.2 MiB/s
xxHash3 | 0.68081 % | 5.07812 % | 40.927 % | 3724.458 | 1252.9 MiB/s
gxHash | 0.72160 % | 5.41992 % | 40.801 % | 3604.803 | 1151.7 MiB/s
StringZilla | 0.72320 % | 5.51758 % | 40.779 % | 3643.094 | 383.5 MiB/s
MurMur3 | 0.73939 % | 5.56641 % | 41.044 % | 3618.672 | 665.4 MiB/s
FxHash | 4.33815 % | 6.92558 % | 40.986 % | 3699.077 | 1455.5 MiB/s
Crc32 | 19.01042 % | 37.50000 % | 41.784 % | 1952.957 | 541.2 MiB/s
RabinKarp32 | 50.00000 % | 50.00000 % | 38.362 % | 1991.701 | 499.3 MiB/s
Hash Quality Analysis - Short N-grams (9-32 bytes):
Function | Avg.Bias | Worst.Bias | Chi² | Throughput
-------------+------------+------------+----------+--------------
FoldHash | 0.01548 % | 0.01775 % | 1004.944 | 4167.8 MiB/s
SipHash | 0.01549 % | 0.01984 % | 1035.135 | 1320.2 MiB/s
gxHash | 0.01556 % | 0.02085 % | 1015.467 | 3303.8 MiB/s
StringZilla | 0.01561 % | 0.02265 % | 1008.633 | 1373.0 MiB/s
xxHash3 | 0.01591 % | 0.02397 % | 1034.249 | 4284.3 MiB/s
aHash | 0.01633 % | 0.02277 % | 1013.296 | 3382.3 MiB/s
SeaHash | 0.01662 % | 0.02079 % | 1024.785 | 1515.0 MiB/s
FarmHash | 0.01663 % | 0.02067 % | 1006.103 | 2112.6 MiB/s
Blake3 | 0.01671 % | 0.02295 % | 1027.414 | 210.7 MiB/s
MurMur3 | 0.01769 % | 0.02418 % | 1017.503 | 1639.7 MiB/s
FxHash | 1.20438 % | 5.16469 % | 1013.976 | 3137.3 MiB/s
Crc32 | 14.78365 % | 20.31250 % | 1025.051 | 604.4 MiB/s
RabinKarp32 | 50.00000 % | 50.00000 % | 1011.610 | 449.9 MiB/s
Hash Quality Analysis - Long N-grams (> 32 bytes):
Function | Avg.Bias | Worst.Bias | Chi² | Throughput
-------------+------------+------------+----------+---------------
MurMur3 | 0.01533 % | 0.02026 % | 1029.882 | 5501.8 MiB/s
SeaHash | 0.01546 % | 0.01758 % | 1057.807 | 5036.2 MiB/s
FxHash | 0.01572 % | 0.02011 % | 1036.177 | 14481.3 MiB/s
gxHash | 0.01585 % | 0.02026 % | 1058.183 | 17059.8 MiB/s
StringZilla | 0.01595 % | 0.01927 % | 1062.383 | 6626.5 MiB/s
aHash | 0.01596 % | 0.02240 % | 1017.026 | 15625.0 MiB/s
Blake3 | 0.01606 % | 0.02107 % | 1032.596 | 976.4 MiB/s
FarmHash | 0.01653 % | 0.02255 % | 1026.637 | 7930.4 MiB/s
SipHash | 0.01669 % | 0.01917 % | 1017.455 | 3826.8 MiB/s
FoldHash | 0.01684 % | 0.02183 % | 1021.117 | 16363.5 MiB/s
xxHash3 | 0.01798 % | 0.02565 % | 1010.240 | 11567.5 MiB/s
Crc32 | 14.06250 % | 18.75000 % | 1024.427 | 4078.8 MiB/s
RabinKarp32 | 50.00000 % | 50.00000 % | 1016.969 | 325.6 MiB/s
```
The results in your README also showed that `gxhash` was 10x slower than `StringZilla`.
```bash
Function | Avg.Bias | Worst.Bias | Integral ⨳ | Chi² | Throughput
-------------+------------+------------+------------+----------+---------------
gxHash | 0.21399 % | 5.41992 % | 41.964 % | 1988.415 | 1020.3 MiB/s
StringZilla | 0.21524 % | 5.51758 % | 41.932 % | 1996.037 | 10994.1 MiB/s
```
But we can see that it's actually the other way around, and this wasn't even with the `hybrid` feature enabled.
```bash
Function | Avg.Bias | Worst.Bias | Integral ⨳ | Chi² | Throughput
-------------+------------+------------+------------+----------+---------------
gxHash | 0.22482 % | 5.41992 % | 40.801 % | 1792.170 | 9798.4 MiB/s
StringZilla | 0.22534 % | 5.51758 % | 40.779 % | 1801.158 | 3784.7 MiB/s
```
I understand that this isn't a throughput benchmark suite, but I think it's misleading to keep the column here.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the benchmark described in the README with 1 million samples, including a run with the hybrid feature enabled, and compare its output with the published tables. Trace how the README results and throughput column are produced; done means the documented results are reproducible or the misleading throughput presentation is corrected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- documentation, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100