LimitPushdown can mistake per-partition fetch for a global limit
- Lingua principale
- Rust
- Stelle
- 9.3k
- Fork
- 2.4k
- Merge medio
- 3g 11h
- PR unite (30g)
- 360
Descrizione
## Describe the bug
`LimitPushdown` can treat an operator-level `fetch` on a multi-partition plan as
if it satisfied a global `LIMIT`.
For operators that do not combine partitions, `ExecutionPlan::fetch()` limits
each output partition independently. Removing the remaining global enforcement
can therefore return up to `fetch * partition_count` rows. Related state
transitions can also place `GlobalLimitExec` directly above multiple partitions
for OFFSET-only plans or combine a per-partition `LocalLimitExec` fetch with a
pending global fetch.
## To Reproduce
Optimize a physical plan shaped like:
```text
GlobalLimitExec: skip=0, fetch=5
TestScan: partitions=2, supports_fetch=true
```
The current rule produces:
```text
TestScan: partitions=2, fetch=5
```
Each partition may emit five rows, so the plan can return ten rows even though
the query has a global limit of five.
The same scope mismatch appears when the child already has a smaller
per-partition fetch, with OFFSET-only, and when a pending global limit reaches a
multi-partition `LocalLimitExec`.
## Expected behavior
A fetch on a multi-partition operator should remain only an early-stop hint.
`LimitPushdown` should retain or create a single-partition enforcement boundary
(`CoalescePartitionsExec` or `SortPreservingMergeExec`) before considering the
global requirement satisfied. Global and per-partition local fetches should not
be combined before that boundary.
## Additional context
The standard physical optimizer pipeline often inserts a partition-combining
boundary before `GlobalLimitExec`, which can hide this problem. It is observable
when optimizing physical extension/distributed plan shapes and when the rule
itself removes or moves limit nodes. The fix can preserve per-partition fetches
for early termination while retaining the global cap.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia tracciando la regola LimitPushdown e il comportamento di ExecutionPlan::fetch() intorno a GlobalLimitExec e LocalLimitExec. Riproduci la struttura di TestScan a due partizioni descritta nell’issue, quindi esamina come CoalescePartitionsExec e SortPreservingMergeExec influiscono sul limite. Il lavoro è completo quando i casi di limite globale, di solo OFFSET e di limite in sospeso mantengono un confine di applicazione a partizione singola, mentre i fetch per partizione restano suggerimenti di arresto anticipato.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- rust
- Ambito
- databases, distributed-systems
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 48/100