apache / apache/pinot

Presto connector is performing full table scan on IS NULL predicate

Open
#7,586 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

presto-connector is doing full table scan on `IS NULL` predicate.

Here is the query example:
`SELECT tool_version, tool_name, feature_type, operation_type_name
FROM map_labeling_logs
WHERE tool_name is null`

Here is the output of EXPLAIN query output:

`- Output[tool_version, tool_name, feature_type, operation_type_name] => [tool_version:varchar, tool_name:integer, feature_type:integer, operation_type_name:varchar]
Estimates: {rows: ? (?), cpu: ?, memory: 0.00, network: ?}
- RemoteStreamingExchange[GATHER] => [operation_type_name:varchar, tool_version:varchar, tool_name:integer, feature_type:integer]
Estimates: {rows: ? (?), cpu: ?, memory: 0.00, network: ?}
- ScanFilter[table = TableHandle {connectorId='pinot', connectorHandle='PinotTableHandle{connectorId=pinot, schemaName=default, tableName=map_labeling_logs, isQueryShort=Optional[false], expectedColumnHandles=Optional[[PinotColumnHandle{columnName=operation_type_name, dataType=varchar, type=REGULAR}, PinotColumnHandle{columnName=tool_version, dataType=varchar, type=REGULAR}, PinotColumnHandle{columnName=tool_name, dataType=integer, type=REGULAR}, PinotColumnHandle{columnName=feature_type, dataType=integer, type=REGULAR}]], pinotQuery=Optional[GeneratedPinotQuery{query=SELECT operation_type_name, tool_version, tool_name, feature_type FROM map_labeling_logs__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 2147483647, format=SQL, table=map_labeling_logs, expectedColumnIndices=[], groupByClauses=0, haveFilter=false, isQueryShort=false}]}', layout='Optional[PinotTableHandle{connectorId=pinot, schemaName=default, tableName=map_labeling_logs, isQueryShort=Optional[false], expectedColumnHandles=Optional[[PinotColumnHandle{columnName=operation_type_name, dataType=varchar, type=REGULAR}, PinotColumnHandle{columnName=tool_version, dataType=varchar, type=REGULAR}, PinotColumnHandle{columnName=tool_name, dataType=integer, type=REGULAR}, PinotColumnHandle{columnName=feature_type, dataType=integer, type=REGULAR}]], pinotQuery=Optional[GeneratedPinotQuery{query=SELECT operation_type_name, tool_version, tool_name, feature_type FROM map_labeling_logs__TABLE_NAME_SUFFIX_TEMPLATE____TIME_BOUNDARY_FILTER_TEMPLATE__ LIMIT 2147483647, format=SQL, table=map_labeling_logs, expectedColumnIndices=[], groupByClauses=0, haveFilter=false, isQueryShort=false}]}]'}, filterPredicate = IS_NULL(tool_name)] => [operation_type_name:varchar, tool_version:varchar, tool_name:integer, feature_type:integer]
Estimates: {rows: ? (?), cpu: ?, memory: 0.00, network: 0.00}/{rows: ? (?), cpu: ?, memory: 0.00, network: 0.00}
tool_version := PinotColumnHandle{columnName=tool_version, dataType=varchar, type=REGULAR}
tool_name := PinotColumnHandle{columnName=tool_name, dataType=integer, type=REGULAR}
operation_type_name := PinotColumnHandle{columnName=operation_type_name, dataType=varchar, type=REGULAR}
feature_type := PinotColumnHandle{columnName=feature_type, dataType=integer, type=REGULAR}`

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Presto connector's handling of the IS_NULL(tool_name) filter and the PinotTableHandle or GeneratedPinotQuery shown in EXPLAIN. Reproduce the query and verify that predicate pushdown produces a filtered Pinot query instead of a full table scan; completion should include regression coverage for this case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.