Support multiple columns in distinct count aggregations
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Currently these [Pinot aggregations](https://docs.pinot.apache.org/users/user-guide-query/supported-aggregations) only work with a single column:
- DISTINCTCOUNT
- DISTINCTCOUNTHLL
- DISTINCTCOUNTRAWHLL
- DistinctCountThetaSketch
- DistinctCountRawThetaSketch
- DISTINCTCOUNTMV
- DISTINCTCOUNTHLLMV
- DISTINCTCOUNTRAWHLLMV
This becomes a problem when you need to get the count of the total number of groups from an aggregation (e.g. to support deeper paging in the dashboard UI), and more than one column is being used for grouping. For example, the query `select advertiser,publisher,sum(adSpend) from table group by advertiser,publisher order by sum(adSpend) desc limit 1000` is using two columns (`advertiser,publisher`) for grouping.
The current workaround is do use `concat` to build a single key, e.g. `select distinctcounthll(concat(advertiser, publisher, '|')) from table`, but that suffers from the performance penalty of using the `concat` scalar UDF.
Contributor guide
Research direction
Start with the linked Pinot supported-aggregations documentation and trace the DISTINCTCOUNT family, including the MV variants. Use the example grouped query and its concat workaround to understand the required behavior. Done means the listed distinct-count aggregations accept multiple grouping columns without relying on the concat scalar UDF.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100