apache / apache/pinot

In ForwardIndexHandler, when converting raw to dictionary encoding, use the stats from ColumnStatistics

Open
#10,508 4 comments 0 reactions 1 assignee Claimed by @vvivekiyer View on GitHub
bug
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

When a column is configured as raw in a real-time table, the committed segment will always have the column metadata with:
- isSorted: false
- cardinality: Integer.MIN_VALUE

Because of this behavior, when converting raw column to dictionary encoded column, we shouldn't rely on these 2 stats from the column metadata. Instead, we should collect the stats again, and read these 2 stats from the `ColumnStatistics` (stats collector).

Currently for a committed segment, even if the column values are sorted, we won't generate sorted index for it because the column metadata always mark it as unsorted.

Related to #10497

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.