Azure / Azure/azure-cli

Investigate security test flakiness

Open
#9,406 2 comments 0 reactions 0 assignees View on GitHub
act-codegen-extensibility-squad Security Service Attention Test Framework
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

The test `test_security_alerts` is behaving flaky on ADO. Possibly related to #9385.

Occasional ADO stack trace:
```
======================================================================
FAIL: test_security_alerts (azure.cli.command_modules.security.tests.latest.test_alerts_scenario.SecurityCenterAlertsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/base.py", line 251, in _in_process_execute
self.exit_code = cli_ctx.invoke(shlex.split(command), out_file=stdout_buf) or 0
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/knack/cli.py", line 218, in invoke
exit_code = self.exception_handler(ex)
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 98, in exception_handler
return handle_exception(ex)
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/patches.py", line 33, in _handle_main_exception
raise ex
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/knack/cli.py", line 206, in invoke
cmd_result = self.invocation.execute(args)
vcr.errors.CannotOverwriteExistingCassetteException: No match for the request () was found. Can't overwrite existing cassette ('/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/command_modules/security/tests/latest/recordings/test_security_alerts.yaml') in your current record mode ('once').

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure_devtools/scenario_tests/decorators.py", line 40, in _preparer_wrapper
fn(test_class_instance, **kwargs)
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/command_modules/security/tests/latest/test_alerts_scenario.py", line 16, in test_security_alerts
alerts = self.cmd('az security alert list').get_output_in_json()
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/base.py", line 161, in cmd
return execute(self.cli_ctx, command, expect_failure=expect_failure).assert_with_checks(checks)
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/base.py", line 200, in __init__
self._in_process_execute(cli_ctx, command, expect_failure=expect_failure)
File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/base.py", line 256, in _in_process_execute
raise AssertionError(ex)
AssertionError: No match for the request () was found. Can't overwrite existing cassette ('/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/command_modules/security/tests/latest/recordings/test_security_alerts.yaml') in your current record mode ('once').
```

When run locally, these tests pass consistently.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.