Sorted index on a non-dictionary-encoded column results in full segment scan
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Defining a `sorted index` on a `non-dictionary-encoded` column causes Pinot to fully scan the content of segments (no inverted index is utilized).
`user` table config:
```
"noDictionaryColumns": [
"id",
],
"sortedColumn": [
"id"
],
```
Example query:
`EXPLAIN PLAN FOR select id, "home.state" from user where id IN ('some_id')`
Output:

Created another user table with a _sorted index_ defined over id which is _dictionary-encoded_.
`user_id_dict_encoded` table config:
```
"noDictionaryColumns": [
],
"sortedColumn": [
"id"
],
```
Example query:
`EXPLAIN PLAN FOR select id, "home.state" from user_id_dict_encoded where id IN ('some_id')`
Output:

@mayankshriv found the root cause of the issue.
Contributor guide
Research direction
Start with the two table configurations and the EXPLAIN PLAN queries in this issue, comparing the non-dictionary-encoded and dictionary-encoded sorted-column cases. Investigate the root cause identified by @mayankshriv and trace why the sorted index is not used for the first case. Done means the non-dictionary-encoded query no longer performs a full segment scan and the plan reflects sorted-index use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100