apache / apache/pinot

[multistage][bug] MV validator issues on new engine

Open
#11,677 0 comments 0 reactions 0 assignees View on GitHub
bug multi-stage
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

several MV validator issues are found

---
exception is not thrown
---

```
explain plan for
select MAXMV(a.mv) from a join b on a.id = b.id limit 10
```
returns
```
Execution Plan
LogicalSort(fetch=[10])
PinotLogicalSortExchange(distribution=[hash], collation=[[]], isSortOnSender=[false], isSortOnReceiver=[false])
LogicalAggregate(group=[{}], agg#0=[MAXMV($0)])
PinotLogicalExchange(distribution=[hash])
LogicalAggregate(group=[{}], agg#0=[MAXMV($1)])
LogicalJoin(condition=[=($0, $2)], joinType=[inner])
PinotLogicalExchange(distribution=[hash[0]])
LogicalProject(id=[$6], mv=[$7])
LogicalTableScan(table=[[a]])
PinotLogicalExchange(distribution=[hash[0]])
LogicalProject(id=[$6])
LogicalTableScan(table=[[b]])
```
but it should failed on MAXMV($1) aggregate b/c it is not supported

---
wrong exception is thrown NO.1
---

```
explain plan for
select MAX(arrayToMv(a.mv)) from a join b on a.id = b.id limit 10
```
seems to throw the exception:
```
Function 'ArrayToMv' is not supported in PROJECT Leaf Stage
org.apache.pinot.query.planner.validation.ArrayToMvValidationVisitor.visitProject(ArrayToMvValidationVisitor.java:99)
org.apache.pinot.query.planner.validation.ArrayToMvValidationVisitor.visitProject(ArrayToMvValidationVisitor.java:43)
org.apache.pinot.query.planner.plannode.ProjectNode.visit(ProjectNode.java:54)
org.apache.pinot.query.planner.validation.ArrayToMvValidationVisitor.lambda$visitMailboxSend$3(ArrayToMvValidationVisitor.java:72)
```
but it is not suppose to be after PROJECT pushdown:
```
Execution Plan
LogicalSort(fetch=[10])
PinotLogicalSortExchange(distribution=[hash], collation=[[]], isSortOnSender=[false], isSortOnReceiver=[false])
LogicalAggregate(group=[{}], agg#0=[MAX($0)])
PinotLogicalExchange(distribution=[hash])
LogicalAggregate(group=[{}], agg#0=[MAX($1)])
LogicalJoin(condition=[=($0, $2)], joinType=[inner])
PinotLogicalExchange(distribution=[hash[0]])
LogicalProject(id=[$6], $f0=[ARRAYTOMV($7)])
LogicalTableScan(table=[[a]])
PinotLogicalExchange(distribution=[hash[0]])
LogicalProject(id=[$6])
LogicalTableScan(table=[[b]])
```

---
wrong exception is thrown NO.2
---

```
explain plan for
select /*+ joinOptions(join_strategy = 'dynamic_broadcast') */
MAX(arrayToMv(a.label_ids))
from github_2 a
where a.id IN (select id FROM github_events b)
```
generates the following plan
```
LogicalAggregate(group=[{}], agg#0=[MAX($0)])
PinotLogicalExchange(distribution=[hash])
LogicalAggregate(group=[{}], agg#0=[MAX($1)])
PinotLogicalExchange(distribution=[hash[0]])
LogicalJoin(condition=[=($0, $8)], joinType=[semi])
LogicalProject(id=[$6], $f0=[ARRAYTOMV($7)])
LogicalTableScan(table=[[a]])
PinotLogicalExchange(distribution=[broadcast], relExchangeType=[PIPELINE_BREAKER])
LogicalTableScan(table=[[b]])
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the three EXPLAIN PLAN cases in the issue and start with the planner validation path, especially ArrayToMvValidationVisitor.java and its visitProject method. Trace how MAXMV and ARRAYTOMV are handled after joins, aggregate planning, and project pushdown. Done means the unsupported aggregate case is rejected and the two ARRAYTOMV cases produce the intended validation behavior rather than the reported wrong exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.