Disable Dictionaries for High Cardinality MV Fields
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
We have a MV columns called `tags__VALUES` in one of our tables. It is a very high cardinality columns. Here's some data from an existing segment:
```
column.tags__VALUES.cardinality = 776088
column.tags__VALUES.totalDocs = 4105983
column.tags__VALUES.dataType = STRING
column.tags__VALUES.bitsPerElement = 20
column.tags__VALUES.lengthOfEachEntry = 512
column.tags__VALUES.columnType = DIMENSION
column.tags__VALUES.isSorted = false
column.tags__VALUES.hasNullValue = false
column.tags__VALUES.hasDictionary = true
column.tags__VALUES.textIndexType = NONE
column.tags__VALUES.hasInvertedIndex = true
column.tags__VALUES.hasFSTIndex = false
column.tags__VALUES.hasJsonIndex = false
column.tags__VALUES.isSingleValues = false
column.tags__VALUES.maxNumberOfMultiValues = 61
column.tags__VALUES.totalNumberOfEntries = 35572346
column.tags__VALUES.isAutoGenerated = false
column.tags__VALUES.minValue =
column.tags__VALUES.maxValue = {id}/reset
column.tags__VALUES.defaultNullValue =
```
AND
```
tags__VALUES.dictionary.startOffset = 294056019
tags__VALUES.dictionary.size = 397357064
tags__VALUES.forward_index.startOffset = 691413083
tags__VALUES.forward_index.size = 93441573
```
For `tags__VALUES`, then dictionary size is ~ 397M. The total size of this segment is 813M, so the dict. itself is occupying around 49% of the size. This certainly doesn't look very optimal.
Docs themselves [suggest](https://docs.pinot.apache.org/basics/indexing/forward-index#:~:text=On%20the%20other,for%20a%20column.) that dict. encoded forward indices are most helpful when the cardinality of the column is low. But this is certainly not the case her.
Not sure if this'll violate any invariant on segments/indexes, but if we can turn off dict. compression for such MV columns, that'd be helpful.
Note: From this [thread](https://apache-pinot.slack.com/archives/C011C9JHN7R/p1650365374544839), I remember Pinot didn't let me create an inverted index on this column while also disabling dict. on it.
Contributor guide
Assessment
This issue has not been assessed yet.