apache / apache/druid

Use fastutil's LongSet in `Sink.dedupSet`

Open
#8,501 0 comments 0 reactions 0 assignees View on GitHub
Area - Streaming Ingestion Contributions Welcome Performance
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

FYI @kaijianding

Also, consider using a proven (SMHasher-wise) hashing algorithm such as xxHash (implemented e. g. in https://github.com/OpenHFT/Zero-Allocation-Hashing) rather than BKDRHash, which seems like a marginal improvement (if improvement at all) over the String's default hash code (the only difference is using 131 instead of 31 as a multiplication constant). While String's default hash function is widely considered a bad hashing algorithm.

Related: #6861

Contributor guide

Open the contributing guide

Research direction

Start by locating the Sink.dedupSet entry point and inspect how it currently stores and hashes values. Read related issue #6861, then determine whether the requested change is limited to using fastutil's LongSet or also includes the xxHash/BKDRHash concern. Done means the agreed scope is implemented and deduplication behavior remains covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.