Investigate security test flakiness
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 3.5k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 60
Description
The test `test_security_alerts` is behaving flaky on ADO. Possibly related to #9385.
Occasional ADO stack trace:
```
======================================================================
FAIL: test_security_alerts (azure.cli.command_modules.security.tests.latest.test_alerts_scenario.SecurityCenterAlertsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/base.py", line 251, in _in_process_execute
self.exit_code = cli_ctx.invoke(shlex.split(command), out_file=stdout_buf) or 0
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/knack/cli.py", line 218, in invoke
exit_code = self.exception_handler(ex)
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 98, in exception_handler
return handle_exception(ex)
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/patches.py", line 33, in _handle_main_exception
raise ex
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/knack/cli.py", line 206, in invoke
cmd_result = self.invocation.execute(args)
vcr.errors.CannotOverwriteExistingCassetteException: No match for the request () was found. Can't overwrite existing cassette ('/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/command_modules/security/tests/latest/recordings/test_security_alerts.yaml') in your current record mode ('once').
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure_devtools/scenario_tests/decorators.py", line 40, in _preparer_wrapper
fn(test_class_instance, **kwargs)
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/command_modules/security/tests/latest/test_alerts_scenario.py", line 16, in test_security_alerts
alerts = self.cmd('az security alert list').get_output_in_json()
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/base.py", line 161, in cmd
return execute(self.cli_ctx, command, expect_failure=expect_failure).assert_with_checks(checks)
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/base.py", line 200, in __init__
self._in_process_execute(cli_ctx, command, expect_failure=expect_failure)
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/base.py", line 256, in _in_process_execute
raise AssertionError(ex)
AssertionError: No match for the request () was found. Can't overwrite existing cassette ('/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/command_modules/security/tests/latest/recordings/test_security_alerts.yaml') in your current record mode ('once').
```
When run locally, these tests pass consistently.
Contributor guide
Assessment
This issue has not been assessed yet.