Azure / Azure/azure-cli

Live test broken with Python 3.11

Open
#27,641 2 comments 0 reactions 2 assignees Claimed by @jiasli View on GitHub
Azure CLI Team bug Test Test Framework
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

https://github.com/Azure/azure-cli/pull/25055 uses `self._outcome.errors` to check the test result:

https://github.com/Azure/azure-cli/blob/674625c388b36ad684e3531bec2c67cde6e00018/src/azure-cli-testsdk/azure/cli/testsdk/scenario_tests/base.py#L166

However, Python 3.11 made a change in https://github.com/python/cpython/pull/28180 that no longer buffers the errors in `_Outcome`. The errors are immediately sent to `result` and `errors` attribute is removed from `_Outcome`, causing failure during Azure CLI's live run:

```py
> azdev test test_role_assignment_scenario --live
...
self =
args = ()

def _save_recording_file(self, *args): # pylint: disable=unused-argument
if self.in_recording and self.cassette.dirty:
self.cassette._save() # pylint: disable=protected-access
> if self._outcome.errors or self._outcome.skipped: # pylint: disable=protected-access
E AttributeError: '_Outcome' object has no attribute 'errors'

src\azure-cli-testsdk\azure\cli\testsdk\scenario_tests\base.py:166: AttributeError
```

https://stackoverflow.com/a/39606065/2199657 suggests to check `self._outcome.result.errors` and claims this approach is compatible with `pytest`, but it is actually not. When invoked by `pytest`, `_pytest.unittest.TestCaseFunction` is passed to `unittest.case.TestCase.__call__` as `result`. As `_pytest.unittest.TestCaseFunction` doesn't have `errors` attribute, `pytest` fails:

```
E AttributeError: 'TestCaseFunction' object has no attribute 'errors'
```

`unittest.case._addError` calls `addError` on the `result` object, making retrieving the errors from `result` test-runner-dependent: https://github.com/python/cpython/blame/2258d6cfa2fcd0c290328e81b68aff25abf8c3d8/Lib/unittest/case.py#L74, https://github.com/python/cpython/blame/2258d6cfa2fcd0c290328e81b68aff25abf8c3d8/Lib/unittest/case.py#L99

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.