Run DataFusion benchmarks regularly and track performance history over time
- Lingua principale
- Rust
- Stelle
- 9.3k
- Fork
- 2.4k
- Merge medio
- 3g 11h
- PR unite (30g)
- 360
Descrizione
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
As we make changes to DataFusion, some changes impact performance and some do not. Right now we mostly rely on reviewers to judge when a change could make an impact on performance, and if so run the appropriate benchmarks.
This means that
1. We may miss some performance regressions (such as https://github.com/apache/arrow-datafusion/issues/5325)
2. Since the benchmarks are not run regularly it is hard to know how to interpret results, and some seem to have bitrotted over time
3. The wide variety of available benchmarks (e.g. https://github.com/apache/arrow-datafusion/issues/5502) makes it hard to know which ones to run and how to determine if performance has improved or regressed for particular changes
**Describe the solution you'd like**
I would like
1. A system that runs DataFusion benchmarks regularly on main
2. Some automated way to see if a particular PR has improved or regressed performance
3. Bonus: a webpage that shows performance over time. Databend has a great example https://perf.databend.rs/
**Suggestion**
I believe conbench, https://conbench.ursa.dev/, which is partially integrated into the repo already, is intended for exactly this usecase. Using conbench would be nice as it appears to be actively maintained and has resources and is already hosted
The integration is https://github.com/apache/arrow-datafusion/tree/main/conbench and was added in https://github.com/apache/arrow-datafusion/pull/1791 by @dianaclarke
You can see its integration as it posts comments on PRs after merge such as https://github.com/apache/arrow-datafusion/pull/5476#issuecomment-1456748444
**Describe alternatives you've considered**
We could use existing timeseries databases and visualizations like grafana to visualize the information
**Additional context**
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia leggendo l'integrazione conbench/ esistente e l'esempio collegato di commento di PR per capire cosa è già supportato. Definisci come eseguire regolarmente i benchmark su main e come identificare una regressione o un miglioramento delle prestazioni di un PR. Il lavoro è completato quando sono disponibili uno storico pianificato dei benchmark e un confronto automatizzato dei PR; la pagina web delle prestazioni è un bonus facoltativo.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- rust
- Ambito
- ci-cd, performance
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Attiva
- Chiarezza
- Da chiarire
- Idoneità per principianti
- 35/100