apache / apache/pinot

[multistage] physical optimizer lookup join support

Open
#17,961 4 comments 0 reactions 1 assignee Claimed by @dang-stripe View on GitHub
mse-physical-optimizer multi-stage stale
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

While testing the physical optimizer, we get this error when trying to invoke a lookup join:

Error log:
```
"message": "Received 1 error from stage N on Server_1234: Right input must be leaf operator",
```

Query:
```
WITH "source_1" AS (
SELECT
'v1' AS "column_1",
1234567890 AS "column_2"
FROM "table_3"
LIMIT 1
)
SELECT
/*+ "joinOptions"("join_strategy" = 'lookup') */
a."column_1",
a."column_2",
b."column_3"
FROM "source_1" a
INNER JOIN "table_2" b
ON b."column_1" = a."column_1"
AND b."column_2" = a."column_2"
```

Claude's analysis:
> The V2 physical optimizer (usePhysicalOptimizer=true) does not correctly handle lookup joins. When a query uses /*+ joinOptions(join_strategy='lookup') */, V2 inserts a BROADCAST exchange on the right side (dim table), which splits the dim table scan into a separate fragment from the join. At runtime, LookupJoinOperator requires rightInput instanceof LeafOperator but receives a MailboxReceiveOperator instead, throwing "Right input must be leaf operator"

cc @ankitsultana @suvodeep-pyne

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.