apache / apache/pinot

[multistage] support additional SQL feature tracker

Open
#9,223 1 comment 4 reactions 0 assignees View on GitHub
multi-stage
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Many SQL features are not yet supported in multistage engine, because one of the two followings:
- stage plan node in pinot-query-planner hasn't been implemented yet.
- operator in pinot-query-runtime hasn't been implemented yet

Some of the existing SQL features missing that prevents us from fully enabling the IntegrationTestSet
- [x] enable partial hard-coded test set: https://github.com/apache/pinot/pull/9412
- CAST with type alias like `LONG` for `BIGINT` is not supported
- MV column support, instead standard ARRAY/LIST SQL type will be
- [x] enable partial generated test set (https://github.com/apache/pinot/pull/9452)

Some already supported by Pinot currently but not on the new engine.
- [x] IN filter clause (https://github.com/apache/pinot/issues/9345) (https://github.com/apache/pinot/pull/9374/)
- [x] RANGE filter clause (https://github.com/apache/pinot/pull/9445)
- [x] ORDER BY (https://github.com/apache/pinot/pull/9279)
- [x] HAVING (https://github.com/apache/pinot/pull/9274)
- [x] DISTINCT (https://github.com/apache/pinot/pull/9383)
- [x] CAST (https://github.com/apache/pinot/pull/9384)
- [x] LIMIT ``, `` (https://github.com/apache/pinot/pull/9496)
- [x] LIKE (https://github.com/apache/pinot/pull/9654)
- [ ] DISTINCT Aggregation
- [ ] UNION/INTERSECT
- [ ] AGG with FILTER

Some of the features are removing limitations of current engine
- [x] support function invoker for non-standard SQL keywords like DATE_TRUNC (requires rewrite of FunctionRegistry to record not just `# of arguments` but a `DataType[]` as function signature, also might as well use function catalog in calcite) (https://github.com/apache/pinot/pull/9375)
- [x] support inequality join condition as post-join filter (https://github.com/apache/pinot/pull/9328)
- [x] count(*) / count(col) / sum(1) are equivalent (without null support), but the rewrite rules are not clear (https://github.com/apache/pinot/pull/9331)
- [x] support more complex join condition expressions, like
- [x] `a JOIN b USING(col)` (https://github.com/apache/pinot/pull/9373)
- [x] `a JOIN b ON b.col = a.col` (ordering matters as currently the ref indexing is assuming left then right) (https://github.com/apache/pinot/pull/9350)
- [x] support LEFT JOIN (https://github.com/apache/pinot/pull/9446)
- [x] support NULL value (https://github.com/apache/pinot/pull/9427)
- [x] support small RIGHT table with inequality conditions. (https://github.com/apache/pinot/pull/9448)
- [x] support SEMI JOIN https://github.com/apache/pinot/pull/9367
- [ ] support NULL-safe operators

Some of the additional items to support but not yet
- [ ] UNNEST (related https://github.com/apache/pinot/issues/7881)
- [ ] LATERAL
- [ ] OVER aggregation (related: https://github.com/apache/pinot/issues/7213)
- [ ] MATCH_REGOGNIZE

Larger Design initiatives
- [x] #9658
- [ ] Function registry with strong type #9641
- [ ] Function registry for other types like UDAF and UDTF https://github.com/apache/pinot/issues/9641

Contributor guide

Open the contributing guide

Research direction

The tracker identifies missing work in pinot-query-planner and pinot-query-runtime, including DISTINCT aggregation, UNION/INTERSECT, NULL-safe operators, UNNEST, LATERAL, and OVER aggregation. Start by selecting one unchecked feature and reviewing its linked issue or existing integration test coverage. Done means the selected SQL feature is supported by the multistage engine and the relevant IntegrationTestSet coverage can be enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.