Azure / Azure/azure-cli

Add Command level benchmark comparison in CI

Open
#14,176 1 comment 0 reactions 2 assignees Claimed by @haroldrandom View on GitHub
act-platform-engineering-squad CI feature-request performance
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

**Is your feature request related to a problem? Please describe.**
With https://github.com/Azure/azure-cli/pull/14156 merged, we are able to collect statstictic data of performance for every command.

But lack of storing them and visualizing it, and leverage the data to monitor and alert any regression.

**Describe the solution you'd like**
Since measuring a command's performance could cost much time, we could setup a daily-scheduled pipeline to measuring all of them in ADO. Then the data could sent to datafactory to store, and extract statistic data into PowerBI to visualize.

As for monitor and alert, we could setup a job in CI to do incremental performance check on any modified commands of a PR, and compare with the histoical statistic data. If the differencies reach out a limit bar, the CI job would fail.

**Describe alternatives you've considered**
As for alert, we could use Gitub PR comment to warn user instead of fail the job.

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.