Support for Dictionary Based Group-By
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
In a recent discussion with @itschrispeck on some related issue, we wondered if it would make sense to have support for executing Group-By by bypassing DocIdSetOperator, and instead using a Dictionary + Inverted Index.
It could help optimize many use-cases, particularly those where a user wants to run some transform function on a group-by/count query (on low cardinality columns) e.g.
```
select
case
when AirTime < 100 then
'ok'
else 'not-ok'
end as airtime_category,
count(*)
from airlineStats
where AirlineID = 19805
group by airtime_category
limit 10
```
We can also optimize json_extract_scalar queries (assuming user has json index on the column):
```
explain plan for select
count(*),
json_extract_scalar(group_json, '$.group_city', 'STRING')
from meetupRsvpJson
group by json_extract_scalar(group_json, '$.group_city', 'STRING')
limit 1000
```
Creating this issue to test the waters and see how other folks feel about this. If there's support we can follow-up with a design doc with more details. Here's a PoC PR I am using to estimate the perf gains: https://github.com/apache/pinot/pull/11758
Contributor guide
Assessment
This issue has not been assessed yet.