apache / apache/datafusion

Eliminate limit when doing scalar aggregation query

Open
#14,974 8 comments 0 reactions 1 assignee Claimed by @niebayes View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

In scalar aggregation queries, the result is always a single row. As a result, applying a LIMIT operator above the Aggregate operator has no effect unless the limit is set to 0. Therefore, I propose we can safely eliminate unnecessary LIMIT operators in scalar aggregation queries.

For example, given a sql
``` sql
select count(*) from t limit 10
```

The `count` UDF always returns a single row and applying a `limit 10` is definitely unnecessary. However, the execution plan has a top `GlobalLimitExec` operator.
```
+---------------+------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+------------------------------------------------------------------------------+
| logical_plan | Limit: skip=0, fetch=10 |
| | Aggregate: groupBy=[[]], aggr=[[count(Int64(1)) AS count(*)]] |
| | TableScan: information_schema.tables projection=[] |
| physical_plan | GlobalLimitExec: skip=0, fetch=10 |
| | AggregateExec: mode=Final, gby=[], aggr=[count(*)] |
| | CoalescePartitionsExec |
| | AggregateExec: mode=Partial, gby=[], aggr=[count(*)] |
| | RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1 |
| | StreamingTableExec: partition_sizes=1 |
| | |
+---------------+------------------------------------------------------------------------------+
```

### Describe the solution you'd like

I think we can design an optimizer rule (logical or physical ??) , to eliminate such unnecessary LIMIT operators.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.