apache / apache/pinot

Sorted index on a non-dictionary-encoded column results in full segment scan

Open
#9,770 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Defining a `sorted index` on a `non-dictionary-encoded` column causes Pinot to fully scan the content of segments (no inverted index is utilized).

`user` table config:
```
"noDictionaryColumns": [
"id",
],
"sortedColumn": [
"id"
],
```

Example query:
`EXPLAIN PLAN FOR select id, "home.state" from user where id IN ('some_id')`

Output:
image

Created another user table with a _sorted index_ defined over id which is _dictionary-encoded_.

`user_id_dict_encoded` table config:
```
"noDictionaryColumns": [
],
"sortedColumn": [
"id"
],
```

Example query:
`EXPLAIN PLAN FOR select id, "home.state" from user_id_dict_encoded where id IN ('some_id')`

Output:
image

@mayankshriv found the root cause of the issue.

Contributor guide

Open the contributing guide

Research direction

Start with the two table configurations and the EXPLAIN PLAN queries in this issue, comparing the non-dictionary-encoded and dictionary-encoded sorted-column cases. Investigate the root cause identified by @mayankshriv and trace why the sorted index is not used for the first case. Done means the non-dictionary-encoded query no longer performs a full segment scan and the plan reflects sorted-index use.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.