Add Command level benchmark comparison in CI
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 3.5k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 60
Description
**Is your feature request related to a problem? Please describe.**
With https://github.com/Azure/azure-cli/pull/14156 merged, we are able to collect statstictic data of performance for every command.
But lack of storing them and visualizing it, and leverage the data to monitor and alert any regression.
**Describe the solution you'd like**
Since measuring a command's performance could cost much time, we could setup a daily-scheduled pipeline to measuring all of them in ADO. Then the data could sent to datafactory to store, and extract statistic data into PowerBI to visualize.
As for monitor and alert, we could setup a job in CI to do incremental performance check on any modified commands of a PR, and compare with the histoical statistic data. If the differencies reach out a limit bar, the CI job would fail.
**Describe alternatives you've considered**
As for alert, we could use Gitub PR comment to warn user instead of fail the job.
**Additional context**
Contributor guide
Assessment
This issue has not been assessed yet.