Cyan4973 / Cyan4973/xxHash

'extremely

Closed
#1,109 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
C
Stars
11.3k
Forks
912
Avg merge
2h 58m
Merged PRs (30d)
24

Description

fat hash it IS on small input yes. as you say, this is a great hash table hasher.... hashing strings (filenames or so).

it is not fast o larger data in any meaningful way and is _not_ cryptographically strong.
I compile all with avx2 but I merged your 'release' branch too to try avx512 but do'nt think this did much.

tmp (tmpfs),
/bin: 3836 files or so at ~2GB.

tmp:
0-50 u.rnd: 40MiB
51-100: u.rnd: 120MiB
small: 500B
big: 4GiB

bsum/csum are ~optimisations of b3sum upstream.

a file of strings and processing at 6ms cost of the while loop
xxhash128sum 260ms (great)
b*sum; +600ms (slow) [ makes sense to not use cryptohashes on this]

mixed:
#zsh
for f in ./xxh128sum b3sum bsum csum : timer $f /tmp/[0-9]* /usr/bin/**/*(-D.)
xxh128sum (fastest for larger data, and the others are 256 wide so kinda adds up to similar speed, granted)

xxh128sum: 1184ms
b3sum: 542ms
bsum: 555ms
csum: 326ms

4G

xxh128sum: 600ms
b3sum: 125ms
bsum: 119ms
csum 1275ms (single threaded backend,, aka upstream's single threaded C implementation)
csum: 172ms

small (500B), like a small note, < 1 fsblk)
all b*sums ~ 5ms
xxh128sum ~2ms

so , great for hash tables, do not use this as a summer in any meaningful way.

Note, I have a private fork of zstd, zsttd22, I call zoom. xxhash is great for ldm, but not for incoming arbtrarily sized data. And it only protect vs. accidental corruption, not malicious! That is probably all fine and well in a datacentre like FBs but not for the general public use. (do what I do, b3).

could be an idea to compile time the two, but I feel as an upstream you wouldn't wanna change abi potentially. api remains same. (+28B frame hdr cost but that's it).

Anyway, I will use this for path strings, but not more. Also, the b,csum's hit a throughput of 41+GB/s.

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file, test, or entry point is named. Start by reproducing the listed zsh benchmarks for xxh128sum, b3sum, bsum, and csum on the described inputs, then determine whether a specific xxHash change is intended; the report does not define what done looks like.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, zsh
Domain
performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.