'extremely
- Dominant language
- C
- Stars
- 11.3k
- Forks
- 912
- Avg merge
- 2h 58m
- Merged PRs (30d)
- 24
Description
fat hash it IS on small input yes. as you say, this is a great hash table hasher.... hashing strings (filenames or so).
it is not fast o larger data in any meaningful way and is _not_ cryptographically strong.
I compile all with avx2 but I merged your 'release' branch too to try avx512 but do'nt think this did much.
tmp (tmpfs),
/bin: 3836 files or so at ~2GB.
tmp:
0-50 u.rnd: 40MiB
51-100: u.rnd: 120MiB
small: 500B
big: 4GiB
bsum/csum are ~optimisations of b3sum upstream.
a file of strings and processing at 6ms cost of the while loop
xxhash128sum 260ms (great)
b*sum; +600ms (slow) [ makes sense to not use cryptohashes on this]
mixed:
#zsh
for f in ./xxh128sum b3sum bsum csum : timer $f /tmp/[0-9]* /usr/bin/**/*(-D.)
xxh128sum (fastest for larger data, and the others are 256 wide so kinda adds up to similar speed, granted)
xxh128sum: 1184ms
b3sum: 542ms
bsum: 555ms
csum: 326ms
4G
xxh128sum: 600ms
b3sum: 125ms
bsum: 119ms
csum 1275ms (single threaded backend,, aka upstream's single threaded C implementation)
csum: 172ms
small (500B), like a small note, < 1 fsblk)
all b*sums ~ 5ms
xxh128sum ~2ms
so , great for hash tables, do not use this as a summer in any meaningful way.
Note, I have a private fork of zstd, zsttd22, I call zoom. xxhash is great for ldm, but not for incoming arbtrarily sized data. And it only protect vs. accidental corruption, not malicious! That is probably all fine and well in a datacentre like FBs but not for the general public use. (do what I do, b3).
could be an idea to compile time the two, but I feel as an upstream you wouldn't wanna change abi potentially. api remains same. (+28B frame hdr cost but that's it).
Anyway, I will use this for path strings, but not more. Also, the b,csum's hit a throughput of 41+GB/s.
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file, test, or entry point is named. Start by reproducing the listed zsh benchmarks for xxh128sum, b3sum, bsum, and csum on the described inputs, then determine whether a specific xxHash change is intended; the report does not define what done looks like.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, zsh
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100