In ForwardIndexHandler, when converting raw to dictionary encoding, use the stats from ColumnStatistics
Open
bug
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
When a column is configured as raw in a real-time table, the committed segment will always have the column metadata with:
- isSorted: false
- cardinality: Integer.MIN_VALUE
Because of this behavior, when converting raw column to dictionary encoded column, we shouldn't rely on these 2 stats from the column metadata. Instead, we should collect the stats again, and read these 2 stats from the `ColumnStatistics` (stats collector).
Currently for a committed segment, even if the column values are sorted, we won't generate sorted index for it because the column metadata always mark it as unsorted.
Related to #10497
Contributor guide
Assessment
This issue has not been assessed yet.