varLengthStringDictionary outperforms fixedLengthStringDictionary for `dictionary.indexOf()`
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
https://github.com/apache/pinot/pull/10007
From the test above, `varLengthStringDictionary` outperforms `fixedSizeStringDictionary` for `dictionary.indexOf()` calls and this is not intuitive because varLengthDictionary has 1 extra level of indirection. We can investigate and understand why this is the case.
```
BenchmarkStringVarLengthDictionary.fixedStringDictionaryIndexOf avgt 5 2802.320 ± 308.715 ms/op
BenchmarkStringVarLengthDictionary.fixedStringDictionaryGet avgt 5 213.743 ± 15.349 ms/op
BenchmarkStringVarLengthDictionary.varLengthStringDictionaryIndexOf avgt 5 2079.673 ± 78.193 ms/op
BenchmarkStringVarLengthDictionary.varLengthStringDictionaryGet avgt 5 227.187 ± 13.771 ms/op
```
Contributor guide
Research direction
Start with linked PR #10007 and the BenchmarkStringVarLengthDictionary benchmark results in the issue. Compare the fixed-length and variable-length dictionary implementations around dictionary.indexOf() and dictionary.get(), then rerun the benchmark to confirm the behavior. Done means explaining why the variable-length dictionary is faster for indexOf() despite its extra indirection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100