apache / apache/datafusion

Run DataFusion benchmarks regularly and track performance history over time

Aperta
#5,504 46 commenti 4 reazioni 0 assegnatari Vedi su GitHub
enhancement help wanted performance
Lingua principale
Rust
Stelle
9.3k
Fork
2.4k
Merge medio
3g 11h
PR unite (30g)
360

Descrizione

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
As we make changes to DataFusion, some changes impact performance and some do not. Right now we mostly rely on reviewers to judge when a change could make an impact on performance, and if so run the appropriate benchmarks.

This means that
1. We may miss some performance regressions (such as https://github.com/apache/arrow-datafusion/issues/5325)
2. Since the benchmarks are not run regularly it is hard to know how to interpret results, and some seem to have bitrotted over time
3. The wide variety of available benchmarks (e.g. https://github.com/apache/arrow-datafusion/issues/5502) makes it hard to know which ones to run and how to determine if performance has improved or regressed for particular changes

**Describe the solution you'd like**
I would like
1. A system that runs DataFusion benchmarks regularly on main
2. Some automated way to see if a particular PR has improved or regressed performance
3. Bonus: a webpage that shows performance over time. Databend has a great example https://perf.databend.rs/

**Suggestion**

I believe conbench, https://conbench.ursa.dev/, which is partially integrated into the repo already, is intended for exactly this usecase. Using conbench would be nice as it appears to be actively maintained and has resources and is already hosted

The integration is https://github.com/apache/arrow-datafusion/tree/main/conbench and was added in https://github.com/apache/arrow-datafusion/pull/1791 by @dianaclarke

You can see its integration as it posts comments on PRs after merge such as https://github.com/apache/arrow-datafusion/pull/5476#issuecomment-1456748444

**Describe alternatives you've considered**
We could use existing timeseries databases and visualizations like grafana to visualize the information

**Additional context**

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia leggendo l'integrazione conbench/ esistente e l'esempio collegato di commento di PR per capire cosa è già supportato. Definisci come eseguire regolarmente i benchmark su main e come identificare una regressione o un miglioramento delle prestazioni di un PR. Il lavoro è completato quando sono disponibili uno storico pianificato dei benchmark e un confronto automatizzato dei PR; la pagina web delle prestazioni è un bonus facoltativo.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
rust
Ambito
ci-cd, performance
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Attiva
Chiarezza
Da chiarire
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.