apache / apache/pinot

[multistage] follow up on multi-stage aggregate functions

Open
#11,034 0 comments 0 reactions 0 assignees View on GitHub
feature multi-stage
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Background
===
- this is a follow-up on #10745
- original design doc: https://docs.google.com/document/d/1Us6aBvTpNLMEy0ODo34OgTk73h_LVFFAH6q17689h1M/edit?usp=sharing

Summary
===
The previous iteration of aggregate function refactor allows all pinot v1 aggregate functions to be planned with v2 engine by registering aggregate functions with v1's agg-function type into multiple aggregate function signatures on v2.

This is a follow-up to ensure the plan can be broken down with the variety of aggregate function flavors, specifically
- [x] (P0) support aggregate function with dynamic return type (and operand type) - https://github.com/apache/pinot/pull/11105
- (P2) additionally as an option, we can also make a cleaner function registry mechanism without using Calcite-specific inferencer/type-checker on `AggregationFunctionType`, they are relatively hard to understand. (as a side effect, this is DONE)
- [x] (P0) support aggregate function that takes more than 1 non-literal arguments - https://github.com/apache/pinot/pull/11105
- [x] (P1) support aggregate function with the literal positional argument used during function constructor - https://github.com/apache/pinot/pull/11105
- [x] (P1) support aggregate function with literal string (e.g. thetaSketch + SET operation) but requires rewrite
- [ ] (P1) support filter after agg (see: https://calcite.apache.org/docs/reference.html#aggregate-functions)
- [ ] (P2) support Calcite's AggregateReduceFunctionsRule to generate common expressions before final reduction (such as VAR_POP, VAR_SAMP running together should reduced into common first-order stats with multiple second-order stats)

Furthermore, we want to clean up some of the components we had to introduce during the 2 initial PR, e.g.
- [x] (P0) the return type of the aggregate doesn't conform with calcite schema and thus require some casting in the AggregateOperator - https://github.com/apache/pinot/pull/11068
- [x] (P0) clean up hint-based aggregate node representation (currently very complex and not used the same system between runtime and planning) - https://github.com/apache/pinot/pull/11036
- [ ] (P2) clean up the final-stage agg vs. final-stage project (so that we don't have to differentiate; and can merge with potential project), some examples include
- skew and kurtosis should share the intermediate agg but only compute final via skew or kurtosis functions;
- percentile with different retrieval percentage value should share the same intermediate percentile spectrun and extract final via the specific percentage positions

Contributor guide

Open the contributing guide

Research direction

Start with the unchecked items in this issue, then read follow-up #10745 and the linked design document. The work is done when the remaining multi-stage aggregate-function cases—post-aggregation filters, AggregateReduceFunctionsRule support, and final-stage aggregation/project cleanup—are supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.