apache / apache/pinot

Support for Dictionary Based Group-By

Open
#11,759 3 comments 2 reactions 1 assignee Claimed by @ankitsultana View on GitHub
enhancement feature
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

In a recent discussion with @itschrispeck on some related issue, we wondered if it would make sense to have support for executing Group-By by bypassing DocIdSetOperator, and instead using a Dictionary + Inverted Index.

It could help optimize many use-cases, particularly those where a user wants to run some transform function on a group-by/count query (on low cardinality columns) e.g.

```
select
case
when AirTime < 100 then
'ok'
else 'not-ok'
end as airtime_category,
count(*)
from airlineStats
where AirlineID = 19805
group by airtime_category
limit 10
```

We can also optimize json_extract_scalar queries (assuming user has json index on the column):
```
explain plan for select
count(*),
json_extract_scalar(group_json, '$.group_city', 'STRING')
from meetupRsvpJson
group by json_extract_scalar(group_json, '$.group_city', 'STRING')
limit 1000
```

Creating this issue to test the waters and see how other folks feel about this. If there's support we can follow-up with a design doc with more details. Here's a PoC PR I am using to estimate the perf gains: https://github.com/apache/pinot/pull/11758

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.