LimitPushdown can mistake per-partition fetch for a global limit
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
## Describe the bug
`LimitPushdown` can treat an operator-level `fetch` on a multi-partition plan as
if it satisfied a global `LIMIT`.
For operators that do not combine partitions, `ExecutionPlan::fetch()` limits
each output partition independently. Removing the remaining global enforcement
can therefore return up to `fetch * partition_count` rows. Related state
transitions can also place `GlobalLimitExec` directly above multiple partitions
for OFFSET-only plans or combine a per-partition `LocalLimitExec` fetch with a
pending global fetch.
## To Reproduce
Optimize a physical plan shaped like:
```text
GlobalLimitExec: skip=0, fetch=5
TestScan: partitions=2, supports_fetch=true
```
The current rule produces:
```text
TestScan: partitions=2, fetch=5
```
Each partition may emit five rows, so the plan can return ten rows even though
the query has a global limit of five.
The same scope mismatch appears when the child already has a smaller
per-partition fetch, with OFFSET-only, and when a pending global limit reaches a
multi-partition `LocalLimitExec`.
## Expected behavior
A fetch on a multi-partition operator should remain only an early-stop hint.
`LimitPushdown` should retain or create a single-partition enforcement boundary
(`CoalescePartitionsExec` or `SortPreservingMergeExec`) before considering the
global requirement satisfied. Global and per-partition local fetches should not
be combined before that boundary.
## Additional context
The standard physical optimizer pipeline often inserts a partition-combining
boundary before `GlobalLimitExec`, which can hide this problem. It is observable
when optimizing physical extension/distributed plan shapes and when the rule
itself removes or moves limit nodes. The fix can preserve per-partition fetches
for early termination while retaining the global cap.
Contributor guide
Research direction
Start by tracing the LimitPushdown rule and the ExecutionPlan::fetch() behavior around GlobalLimitExec and LocalLimitExec. Reproduce the two-partition TestScan shape described in the issue, then inspect how CoalescePartitionsExec and SortPreservingMergeExec affect the limit. Done means global, OFFSET-only, and pending-limit cases retain a single-partition enforcement boundary while per-partition fetches remain early-stop hints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100