apache / apache/pinot

Support multiple columns in distinct count aggregations

Open
#6,676 0 comments 4 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently these [Pinot aggregations](https://docs.pinot.apache.org/users/user-guide-query/supported-aggregations) only work with a single column:

- DISTINCTCOUNT
- DISTINCTCOUNTHLL
- DISTINCTCOUNTRAWHLL
- DistinctCountThetaSketch
- DistinctCountRawThetaSketch
- DISTINCTCOUNTMV
- DISTINCTCOUNTHLLMV
- DISTINCTCOUNTRAWHLLMV

This becomes a problem when you need to get the count of the total number of groups from an aggregation (e.g. to support deeper paging in the dashboard UI), and more than one column is being used for grouping. For example, the query `select advertiser,publisher,sum(adSpend) from table group by advertiser,publisher order by sum(adSpend) desc limit 1000` is using two columns (`advertiser,publisher`) for grouping.

The current workaround is do use `concat` to build a single key, e.g. `select distinctcounthll(concat(advertiser, publisher, '|')) from table`, but that suffers from the performance penalty of using the `concat` scalar UDF.

Contributor guide

Open the contributing guide

Research direction

Start with the linked Pinot supported-aggregations documentation and trace the DISTINCTCOUNT family, including the MV variants. Use the example grouped query and its concat workaround to understand the required behavior. Done means the listed distinct-count aggregations accept multiple grouping columns without relying on the concat scalar UDF.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.