[multistage][bug] MV validator issues on new engine
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
several MV validator issues are found
---
exception is not thrown
---
```
explain plan for
select MAXMV(a.mv) from a join b on a.id = b.id limit 10
```
returns
```
Execution Plan
LogicalSort(fetch=[10])
PinotLogicalSortExchange(distribution=[hash], collation=[[]], isSortOnSender=[false], isSortOnReceiver=[false])
LogicalAggregate(group=[{}], agg#0=[MAXMV($0)])
PinotLogicalExchange(distribution=[hash])
LogicalAggregate(group=[{}], agg#0=[MAXMV($1)])
LogicalJoin(condition=[=($0, $2)], joinType=[inner])
PinotLogicalExchange(distribution=[hash[0]])
LogicalProject(id=[$6], mv=[$7])
LogicalTableScan(table=[[a]])
PinotLogicalExchange(distribution=[hash[0]])
LogicalProject(id=[$6])
LogicalTableScan(table=[[b]])
```
but it should failed on MAXMV($1) aggregate b/c it is not supported
---
wrong exception is thrown NO.1
---
```
explain plan for
select MAX(arrayToMv(a.mv)) from a join b on a.id = b.id limit 10
```
seems to throw the exception:
```
Function 'ArrayToMv' is not supported in PROJECT Leaf Stage
org.apache.pinot.query.planner.validation.ArrayToMvValidationVisitor.visitProject(ArrayToMvValidationVisitor.java:99)
org.apache.pinot.query.planner.validation.ArrayToMvValidationVisitor.visitProject(ArrayToMvValidationVisitor.java:43)
org.apache.pinot.query.planner.plannode.ProjectNode.visit(ProjectNode.java:54)
org.apache.pinot.query.planner.validation.ArrayToMvValidationVisitor.lambda$visitMailboxSend$3(ArrayToMvValidationVisitor.java:72)
```
but it is not suppose to be after PROJECT pushdown:
```
Execution Plan
LogicalSort(fetch=[10])
PinotLogicalSortExchange(distribution=[hash], collation=[[]], isSortOnSender=[false], isSortOnReceiver=[false])
LogicalAggregate(group=[{}], agg#0=[MAX($0)])
PinotLogicalExchange(distribution=[hash])
LogicalAggregate(group=[{}], agg#0=[MAX($1)])
LogicalJoin(condition=[=($0, $2)], joinType=[inner])
PinotLogicalExchange(distribution=[hash[0]])
LogicalProject(id=[$6], $f0=[ARRAYTOMV($7)])
LogicalTableScan(table=[[a]])
PinotLogicalExchange(distribution=[hash[0]])
LogicalProject(id=[$6])
LogicalTableScan(table=[[b]])
```
---
wrong exception is thrown NO.2
---
```
explain plan for
select /*+ joinOptions(join_strategy = 'dynamic_broadcast') */
MAX(arrayToMv(a.label_ids))
from github_2 a
where a.id IN (select id FROM github_events b)
```
generates the following plan
```
LogicalAggregate(group=[{}], agg#0=[MAX($0)])
PinotLogicalExchange(distribution=[hash])
LogicalAggregate(group=[{}], agg#0=[MAX($1)])
PinotLogicalExchange(distribution=[hash[0]])
LogicalJoin(condition=[=($0, $8)], joinType=[semi])
LogicalProject(id=[$6], $f0=[ARRAYTOMV($7)])
LogicalTableScan(table=[[a]])
PinotLogicalExchange(distribution=[broadcast], relExchangeType=[PIPELINE_BREAKER])
LogicalTableScan(table=[[b]])
```
Contributor guide
Research direction
Reproduce the three EXPLAIN PLAN cases in the issue and start with the planner validation path, especially ArrayToMvValidationVisitor.java and its visitProject method. Trace how MAXMV and ARRAYTOMV are handled after joins, aggregate planning, and project pushdown. Done means the unsupported aggregate case is rejected and the two ARRAYTOMV cases produce the intended validation behavior rather than the reported wrong exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100