apache / apache/pinot

Disable Dictionaries for High Cardinality MV Fields

Open
#8,750 4 comments 0 reactions 1 assignee Claimed by @siddharthteotia View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

We have a MV columns called `tags__VALUES` in one of our tables. It is a very high cardinality columns. Here's some data from an existing segment:

```
column.tags__VALUES.cardinality = 776088
column.tags__VALUES.totalDocs = 4105983
column.tags__VALUES.dataType = STRING
column.tags__VALUES.bitsPerElement = 20
column.tags__VALUES.lengthOfEachEntry = 512
column.tags__VALUES.columnType = DIMENSION
column.tags__VALUES.isSorted = false
column.tags__VALUES.hasNullValue = false
column.tags__VALUES.hasDictionary = true
column.tags__VALUES.textIndexType = NONE
column.tags__VALUES.hasInvertedIndex = true
column.tags__VALUES.hasFSTIndex = false
column.tags__VALUES.hasJsonIndex = false
column.tags__VALUES.isSingleValues = false
column.tags__VALUES.maxNumberOfMultiValues = 61
column.tags__VALUES.totalNumberOfEntries = 35572346
column.tags__VALUES.isAutoGenerated = false
column.tags__VALUES.minValue =
column.tags__VALUES.maxValue = {id}/reset
column.tags__VALUES.defaultNullValue =
```

AND

```
tags__VALUES.dictionary.startOffset = 294056019
tags__VALUES.dictionary.size = 397357064
tags__VALUES.forward_index.startOffset = 691413083
tags__VALUES.forward_index.size = 93441573
```

For `tags__VALUES`, then dictionary size is ~ 397M. The total size of this segment is 813M, so the dict. itself is occupying around 49% of the size. This certainly doesn't look very optimal.

Docs themselves [suggest](https://docs.pinot.apache.org/basics/indexing/forward-index#:~:text=On%20the%20other,for%20a%20column.) that dict. encoded forward indices are most helpful when the cardinality of the column is low. But this is certainly not the case her.

Not sure if this'll violate any invariant on segments/indexes, but if we can turn off dict. compression for such MV columns, that'd be helpful.

Note: From this [thread](https://apache-pinot.slack.com/archives/C011C9JHN7R/p1650365374544839), I remember Pinot didn't let me create an inverted index on this column while also disabling dict. on it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.