apache / apache/pinot

Provide a way to group by an entire multi value field

Open
#8,353 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently, Pinot explodes multi value columns included in grouping sets. This is desirable for many use cases but differs from how some other sql engines behave. Provide a way to group by an entire multi value field instead of exploding it, such as a transform function.

Here is an example to illustrate:
For a table with a multi value string array column and long column with the following values:
```
['hi', 'hello'], 3
['bye'], 4
['good', 'bye'], 2
```
And the query and results:
```
select string_array_col, count(*) from table group by string_array_col;
```
Results:
```
'hi', 1
'bye', 2
'good', 1
'hello', 1
```

Provide a transform function so that we can group by the entire array, for example:
```
select joinmv(string_array_col), count(*) from table group by joinmv(string_array_col)
['hi', 'hello'], 1
['bye'], 1
['good', 'bye'], 1
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing Pinot's handling of multi-value columns in grouping sets and the existing transform-function entry points. The work is done when a function such as joinmv lets grouping preserve each complete array, producing the three grouped results shown in the issue rather than exploding values.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.