apache / apache/pinot

varLengthStringDictionary outperforms fixedLengthStringDictionary for `dictionary.indexOf()`

Open
#10,008 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

https://github.com/apache/pinot/pull/10007

From the test above, `varLengthStringDictionary` outperforms `fixedSizeStringDictionary` for `dictionary.indexOf()` calls and this is not intuitive because varLengthDictionary has 1 extra level of indirection. We can investigate and understand why this is the case.

```
BenchmarkStringVarLengthDictionary.fixedStringDictionaryIndexOf avgt 5 2802.320 ± 308.715 ms/op
BenchmarkStringVarLengthDictionary.fixedStringDictionaryGet avgt 5 213.743 ± 15.349 ms/op
BenchmarkStringVarLengthDictionary.varLengthStringDictionaryIndexOf avgt 5 2079.673 ± 78.193 ms/op
BenchmarkStringVarLengthDictionary.varLengthStringDictionaryGet avgt 5 227.187 ± 13.771 ms/op
```

Contributor guide

Open the contributing guide

Research direction

Start with linked PR #10007 and the BenchmarkStringVarLengthDictionary benchmark results in the issue. Compare the fixed-length and variable-length dictionary implementations around dictionary.indexOf() and dictionary.get(), then rerun the benchmark to confirm the behavior. Done means explaining why the variable-length dictionary is faster for indexOf() despite its extra indirection.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.