apache / apache/datafusion

[EPIC] Improving cost calculations and cost based optimizations

Aperta
#3,929 8 commenti 3 reazioni 0 assegnatari Vedi su GitHub
enhancement PROPOSAL EPIC
Lingua principale
Rust
Stelle
9.3k
Fork
2.4k
Merge medio
3g 11h
PR unite (30g)
360

Descrizione

Design document: https://docs.google.com/document/d/1M4mmV7KA1LSj-D-WJA338B4ydlm-8A8D5OPuDE5_SD4/edit

This is a meta issue for improving cost calculations and cost-based optimizations in DataFusion. We already have some statistics collected (mainly from the table sources) and there are estimations for statistics by some of the execution plan nodes, and the overall idea is to improve these as well as possible CBOs.

### Main Goals
- Have enough statistics to start nested join optimizations (#3843). This involves being able to estimate the weight of a join side, and do global re-ordering between join sides to minimize the overall cost of parent joins by reducing the output as much as possible at the bottom levels.
- Provide a more reliable static analysis phase for physical execution operators (so that range based pruning/predicate pruning can leverage the existing infrastructure on their implementations)
- What else?

### Work in Progress

- [x] https://github.com/apache/arrow-datafusion/issues/3898
- [x] https://github.com/apache/arrow-datafusion/issues/3845
- [ ] https://github.com/apache/arrow-datafusion/issues/4158
- [ ] https://github.com/apache/arrow-datafusion/issues/4159
- What else?

### Planned
- [ ] Estimating join cardinalities when the underlying table does not have any statistics (https://github.com/apache/arrow-datafusion/issues/3813#issuecomment-1276643214).
- What else?

### Future
- Support for histograms, so better value distribution when working with cardinality estimations / filter selectivity. Currently, none of the providers we use can directly pass it to us, so we either have to take a peek at the data or only expose the API for other services (like ballista) which can actually collect it and pass to us.

P.S.: feel free to update the text directly or let me know (and I can update it myself)

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia dal documento di progettazione collegato, quindi esamina le issue completate e aperte elencate in Work in Progress e Planned, in particolare #3843, #4158, #4159 e #3813. L’issue non ha un singolo punto di ingresso per l’implementazione né un criterio di completamento; il progresso consiste nell’affrontare gli obiettivi elencati di ottimizzazione basata sulle statistiche e sui costi e nell’aggiornare questa meta-issue.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
rust
Ambito
data, databases, performance
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
20/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.