Azure / Azure/azure-cli

"Live Only" test failures should be identified and flagged with high priority

Open
#8,368 2 comments 0 reactions 1 assignee Claimed by @qwordy View on GitHub
feature-request Infrastructure Test Framework
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

**Is your feature request related to a problem? Please describe.**
We have a number of tests that are marked `@live_only` or `LiveScenarioTest` because they cannot be replayed as a recording. In these cases, the nightly live run is the only way to detect a CLI regression. Unfortunately, due to the noisiness of the live test system, regressions are not identified and corrected immediately.

**Describe the solution you'd like**
All data from the live test runs are stored in a database. We can analyze our tests to identify exactly which ones are live only. We should be able to check the daily live test failures and compare it against this list. If a live-only test fails, it should trigger a more elevated response so that it gets immediate attention.

**Describe alternatives you've considered**
Aside from an automated solution, this scenario relies on consistent, dedicated human intervention, which is not realistic.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.