ashvardanian / ashvardanian/HashEvals

Getting extremely different benchmarks from the README

Open
#1 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
10
Forks
1
PR merge metrics
No merged PRs in 30d

Description

I ran 1 million samples as instructed in the README and this was what I got. More importantly, it shows that `StringZilla` is way slower than shown in the README. 10994.1 MiB/s (README) vs 3784.7 MiB/s (actuality)

```bash
Hash Quality Analysis:

Function | Avg.Bias | Worst.Bias | Integral ⨳ | Chi² | Throughput
-------------+------------+------------+------------+----------+---------------
Blake3 | 0.16303 % | 3.75977 % | 41.194 % | 1828.047 | 586.3 MiB/s
SeaHash | 0.18943 % | 4.44336 % | 40.876 % | 1806.709 | 3578.1 MiB/s
aHash | 0.19246 % | 4.54102 % | 40.711 % | 1771.800 | 9364.9 MiB/s
SipHash | 0.20437 % | 4.88281 % | 40.531 % | 1806.103 | 2869.7 MiB/s
FoldHash | 0.20813 % | 4.88281 % | 41.223 % | 1788.240 | 10334.5 MiB/s
FarmHash | 0.21102 % | 5.07812 % | 40.973 % | 1778.461 | 5306.1 MiB/s
xxHash3 | 0.21338 % | 5.07812 % | 40.927 % | 1826.013 | 8549.2 MiB/s
gxHash | 0.22482 % | 5.41992 % | 40.801 % | 1792.170 | 9798.4 MiB/s
StringZilla | 0.22534 % | 5.51758 % | 40.779 % | 1801.158 | 3784.7 MiB/s
MurMur3 | 0.23100 % | 5.56641 % | 41.044 % | 1790.971 | 3905.9 MiB/s
FxHash | 1.86875 % | 6.92558 % | 40.986 % | 1814.495 | 9140.8 MiB/s
Crc32 | 15.87577 % | 37.50000 % | 41.784 % | 1299.847 | 2292.7 MiB/s
RabinKarp32 | 50.00000 % | 50.00000 % | 38.362 % | 1303.198 | 338.5 MiB/s

Hash Quality Analysis - Tiny N-grams (≤ 8 bytes):

Function | Avg.Bias | Worst.Bias | Integral ⨳ | Chi² | Throughput
-------------+------------+------------+------------+----------+--------------
Blake3 | 0.51103 % | 3.75977 % | 41.194 % | 3725.665 | 69.7 MiB/s
SeaHash | 0.60072 % | 4.44336 % | 40.876 % | 3639.012 | 600.1 MiB/s
aHash | 0.61103 % | 4.54102 % | 40.711 % | 3570.448 | 1096.3 MiB/s
SipHash | 0.65208 % | 4.88281 % | 40.531 % | 3650.411 | 541.2 MiB/s
FoldHash | 0.66464 % | 4.88281 % | 41.223 % | 3636.436 | 1175.4 MiB/s
FarmHash | 0.67278 % | 5.07812 % | 40.973 % | 3597.411 | 765.2 MiB/s
xxHash3 | 0.68081 % | 5.07812 % | 40.927 % | 3724.458 | 1252.9 MiB/s
gxHash | 0.72160 % | 5.41992 % | 40.801 % | 3604.803 | 1151.7 MiB/s
StringZilla | 0.72320 % | 5.51758 % | 40.779 % | 3643.094 | 383.5 MiB/s
MurMur3 | 0.73939 % | 5.56641 % | 41.044 % | 3618.672 | 665.4 MiB/s
FxHash | 4.33815 % | 6.92558 % | 40.986 % | 3699.077 | 1455.5 MiB/s
Crc32 | 19.01042 % | 37.50000 % | 41.784 % | 1952.957 | 541.2 MiB/s
RabinKarp32 | 50.00000 % | 50.00000 % | 38.362 % | 1991.701 | 499.3 MiB/s

Hash Quality Analysis - Short N-grams (9-32 bytes):

Function | Avg.Bias | Worst.Bias | Chi² | Throughput
-------------+------------+------------+----------+--------------
FoldHash | 0.01548 % | 0.01775 % | 1004.944 | 4167.8 MiB/s
SipHash | 0.01549 % | 0.01984 % | 1035.135 | 1320.2 MiB/s
gxHash | 0.01556 % | 0.02085 % | 1015.467 | 3303.8 MiB/s
StringZilla | 0.01561 % | 0.02265 % | 1008.633 | 1373.0 MiB/s
xxHash3 | 0.01591 % | 0.02397 % | 1034.249 | 4284.3 MiB/s
aHash | 0.01633 % | 0.02277 % | 1013.296 | 3382.3 MiB/s
SeaHash | 0.01662 % | 0.02079 % | 1024.785 | 1515.0 MiB/s
FarmHash | 0.01663 % | 0.02067 % | 1006.103 | 2112.6 MiB/s
Blake3 | 0.01671 % | 0.02295 % | 1027.414 | 210.7 MiB/s
MurMur3 | 0.01769 % | 0.02418 % | 1017.503 | 1639.7 MiB/s
FxHash | 1.20438 % | 5.16469 % | 1013.976 | 3137.3 MiB/s
Crc32 | 14.78365 % | 20.31250 % | 1025.051 | 604.4 MiB/s
RabinKarp32 | 50.00000 % | 50.00000 % | 1011.610 | 449.9 MiB/s

Hash Quality Analysis - Long N-grams (> 32 bytes):

Function | Avg.Bias | Worst.Bias | Chi² | Throughput
-------------+------------+------------+----------+---------------
MurMur3 | 0.01533 % | 0.02026 % | 1029.882 | 5501.8 MiB/s
SeaHash | 0.01546 % | 0.01758 % | 1057.807 | 5036.2 MiB/s
FxHash | 0.01572 % | 0.02011 % | 1036.177 | 14481.3 MiB/s
gxHash | 0.01585 % | 0.02026 % | 1058.183 | 17059.8 MiB/s
StringZilla | 0.01595 % | 0.01927 % | 1062.383 | 6626.5 MiB/s
aHash | 0.01596 % | 0.02240 % | 1017.026 | 15625.0 MiB/s
Blake3 | 0.01606 % | 0.02107 % | 1032.596 | 976.4 MiB/s
FarmHash | 0.01653 % | 0.02255 % | 1026.637 | 7930.4 MiB/s
SipHash | 0.01669 % | 0.01917 % | 1017.455 | 3826.8 MiB/s
FoldHash | 0.01684 % | 0.02183 % | 1021.117 | 16363.5 MiB/s
xxHash3 | 0.01798 % | 0.02565 % | 1010.240 | 11567.5 MiB/s
Crc32 | 14.06250 % | 18.75000 % | 1024.427 | 4078.8 MiB/s
RabinKarp32 | 50.00000 % | 50.00000 % | 1016.969 | 325.6 MiB/s
```

The results in your README also showed that `gxhash` was 10x slower than `StringZilla`.

```bash
Function | Avg.Bias | Worst.Bias | Integral ⨳ | Chi² | Throughput
-------------+------------+------------+------------+----------+---------------
gxHash | 0.21399 % | 5.41992 % | 41.964 % | 1988.415 | 1020.3 MiB/s
StringZilla | 0.21524 % | 5.51758 % | 41.932 % | 1996.037 | 10994.1 MiB/s
```

But we can see that it's actually the other way around, and this wasn't even with the `hybrid` feature enabled.

```bash
Function | Avg.Bias | Worst.Bias | Integral ⨳ | Chi² | Throughput
-------------+------------+------------+------------+----------+---------------
gxHash | 0.22482 % | 5.41992 % | 40.801 % | 1792.170 | 9798.4 MiB/s
StringZilla | 0.22534 % | 5.51758 % | 40.779 % | 1801.158 | 3784.7 MiB/s
```

I understand that this isn't a throughput benchmark suite, but I think it's misleading to keep the column here.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the benchmark described in the README with 1 million samples, including a run with the hybrid feature enabled, and compare its output with the published tables. Trace how the README results and throughput column are produced; done means the documented results are reproducible or the misleading throughput presentation is corrected.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
documentation, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.