apache / apache/datafusion

LimitPushdown can mistake per-partition fetch for a global limit

Open
#23,799 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

## Describe the bug

`LimitPushdown` can treat an operator-level `fetch` on a multi-partition plan as
if it satisfied a global `LIMIT`.

For operators that do not combine partitions, `ExecutionPlan::fetch()` limits
each output partition independently. Removing the remaining global enforcement
can therefore return up to `fetch * partition_count` rows. Related state
transitions can also place `GlobalLimitExec` directly above multiple partitions
for OFFSET-only plans or combine a per-partition `LocalLimitExec` fetch with a
pending global fetch.

## To Reproduce

Optimize a physical plan shaped like:

```text
GlobalLimitExec: skip=0, fetch=5
TestScan: partitions=2, supports_fetch=true
```

The current rule produces:

```text
TestScan: partitions=2, fetch=5
```

Each partition may emit five rows, so the plan can return ten rows even though
the query has a global limit of five.

The same scope mismatch appears when the child already has a smaller
per-partition fetch, with OFFSET-only, and when a pending global limit reaches a
multi-partition `LocalLimitExec`.

## Expected behavior

A fetch on a multi-partition operator should remain only an early-stop hint.
`LimitPushdown` should retain or create a single-partition enforcement boundary
(`CoalescePartitionsExec` or `SortPreservingMergeExec`) before considering the
global requirement satisfied. Global and per-partition local fetches should not
be combined before that boundary.

## Additional context

The standard physical optimizer pipeline often inserts a partition-combining
boundary before `GlobalLimitExec`, which can hide this problem. It is observable
when optimizing physical extension/distributed plan shapes and when the rule
itself removes or moves limit nodes. The fix can preserve per-partition fetches
for early termination while retaining the global cap.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the LimitPushdown rule and the ExecutionPlan::fetch() behavior around GlobalLimitExec and LocalLimitExec. Reproduce the two-partition TestScan shape described in the issue, then inspect how CoalescePartitionsExec and SortPreservingMergeExec affect the limit. Done means global, OFFSET-only, and pending-limit cases retain a single-partition enforcement boundary while per-partition fetches remain early-stop hints.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.