"Live Only" test failures should be identified and flagged with high priority
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 3.5k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 60
Description
**Is your feature request related to a problem? Please describe.**
We have a number of tests that are marked `@live_only` or `LiveScenarioTest` because they cannot be replayed as a recording. In these cases, the nightly live run is the only way to detect a CLI regression. Unfortunately, due to the noisiness of the live test system, regressions are not identified and corrected immediately.
**Describe the solution you'd like**
All data from the live test runs are stored in a database. We can analyze our tests to identify exactly which ones are live only. We should be able to check the daily live test failures and compare it against this list. If a live-only test fails, it should trigger a more elevated response so that it gets immediate attention.
**Describe alternatives you've considered**
Aside from an automated solution, this scenario relies on consistent, dedicated human intervention, which is not realistic.
Contributor guide
Assessment
This issue has not been assessed yet.