apache / apache/datafusion

[EPIC] Adaptive predicate evaluation

Aperta
#22,883 4 commenti 1 reazione 0 assegnatari Vedi su GitHub
Lingua principale
Rust
Stelle
9.3k
Fork
2.4k
Merge medio
3g 11h
PR unite (30g)
360

Descrizione

This is a tracking ticket for work @adriangb @neilconway and others have discussed in various places. I am putting it into a single ticket so it is easier to track

## Summary

The idea is that the order of applying filters can make a big difference in performance. For example
1. running very selective filters before non selective filters is often a good idea
2. Running filters *during* the parquet scan rather than decoding first then filtering is often a good ida

However in practice, neither of the above is always true:
1. Sometimes selective filters are very expensive to evaluate (e.g. a regexp_match) compared to a relative less expensive filter (e.g. `int8_col = 1`)
2. Sometimes pushing predicates down into the parquet scan actually slows things down (see https://github.com/apache/datafusion/issues/3463)

We have some heuristics for choosing the predicate evaluation order and when to push filters down, but they inevitably get it wrong sometimes (e.g. because the selectivity estimate is wrong, or the selected row pattern is bad).

The idea is to track how effective (and expensive) a predicate actually is at runtime and then adapt the ordering and placement based on that

## Related PRs

- #22144 — [Experiment] Adaptive filter pushdown (umbrella experiment)
- #22234 — `OptionalFilterPhysicalExpr` wrapper + proto (mark a filter as droppable)
- #22235 — Per-conjunct pruning statistics for `PruningPredicate`
- #22236 — `SelectivityTracker` adaptive filter cost model
- #22237 — Adaptive filter pushdown for the parquet scan (integration)
- #21752 — Adaptive filter scheduling for Parquet scans (prior full PR)
- #20363 — (Test) Advanced adaptive filter selectivity evaluation
- #19639 — feat: adaptive filter selectivity tracking for Parquet row filters (closed prototype)
- apache/arrow-rs#9659 — complementary compute-only filter evaluation optimizations

## Related issues

- #15512 — [Epic] Dynamic filtering related items
- #21207 — [DISCUSSION] Future of Dynamic Filters Sync
- #3463 — Enable parquet `filter_pushdown` by default
- https://github.com/apache/arrow-rs/issues/8846 in arrow-rs from @hhhizzz

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Questo è un epic di monitoraggio, non una modifica autonoma. Inizia leggendo le PR correlati #22144, #22234, #22235, #22236 e #22237, insieme all’issue #3463, per comprendere il lavoro esistente sul filtro adattivo. Il lavoro sarà considerato completato quando sarà definito l’ambito di implementazione per l’ordinamento e il posizionamento della valutazione dei predicati a runtime, insieme alla relativa validazione.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
rust
Ambito
data, performance
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Tranquilla
Chiarezza
Da chiarire
Idoneità per principianti
25/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.