allure-framework / allure-framework/allure-python
Add support for coroutine-based concurrency
- Dominant language
- Python
- Stars
- 814
- Forks
- 260
- PR merge metrics
- No merged PRs in 30d
Description
Currently, we don\`t support running allure-python in an async function (e.g. through asyncio event loop or a similar coroutine execution engine).
This is important, because
1. Async-based frameworks and libraries are growing in popularity (`aiohttp`, `aiofile`, `aiopg`, etc.), so we need to support testing asynchronous code
2. E2E tests are heavily IO-bound. The ability to run them concurrently may improve tests performance significantly.
#### Coroutines and contexts
A coroutine in python is created with a coroutine declaration, i.e. a function with the `async` keyword. Inside a function the `await` keyword can be used to wait for an asynchronous operation to complete. A coroutine execution engine can then pause the coroutine and resume execution of another one, increasing performance of IO-bound computations.
Without proper means a coroutine paused inside a context manager may leak unintended global state into another coroutine.
#### Basic illustration
Suppose we have the following abstract code:
```python
async def f():
with change_global_state():
await long_io_operation()
async def g():
use_global_state()
```
If we create two coroutines (one with `f()` and the other one with `g()`) and schedule both coroutines for execution, it is possible that `g` will use the state, not expected to be visible outside `f`.
Below is an example, how both coroutines could be run with `asyncio`:
```python
import asyncio
async def execute_all():
await asyncio.gather(f(), g())
asyncio.run(execute_all())
```
There are two general scenarios we can face while generating allure report. They are described below.
### Scenario 1: running multiple tests concurrently
This scenario requires a pytest plugin capable of executing async tests concurrently, like [pytest-asyncio-cooperative](https://github.com/willemt/pytest-asyncio-cooperative). Unfortunately, this particular plugin currently doesn\`t work with allure-pytest, so the remaining content in this section describes more of a theoretical case, as if the plugin actually works.
If we assume the plugin works, the test may looks like this ([also, see here](https://github.com/allure-framework/allure-python/blob/a86a8f4172681257bc64eb3ff8feeb78592090c2/allure-pytest/test/acceptance/concurrency/async_concurrency_test.py#L51)):
```python
import allure
import asyncio
import pytest
@pytest.fixture(scope="session")
async def fence():
yield asyncio.Event()
@pytest.mark.asyncio_cooperative
async def test_with_async_operation(fence):
with allure.step("Wait for a fence"):
await fence.wait()
@pytest.mark.asyncio_cooperative
async def test_without_async_operation(fence):
with allure.step("Setting a fence"):
fence.set()
```
After `test_with_async_operation` is paused, the event loop starts executing `test_without_async_operation`. The step `"Setting a fence"` is created inside the `"Wait for a fence""` step context, emitting an invalid test result: the first test result contains two nested steps instead of one, while the second one contains no steps whatsoever.
### Scenario 2: running multiple steps concurrently inside one test
This one is easily achieved either with a self-written fixture, or with [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio).
Given we have pytest-asyncio installed and the following code:
```python
import pytest
import allure
import asyncio
@pytest.mark.asyncio
async def test_with_concurrent_steps():
fence = asyncio.Event()
async def run_with_step(name):
with allure.step(name):
await fence.wait()
async def release_fence():
fence.set()
await asyncio.gather(
run_with_step("Step 1"),
run_with_step("Step 2"),
release_fence()
)
```
Note: the steps could've been executed with `asyncio`'s `create_task` function instead with the same effect.
The following `*result.json` is generated (most unrelated fields are omitted):
```json
{
"name": "test_with_concurrent_steps",
"status": "passed",
"steps": [{
"name": "Step 1",
"status": "passed",
"steps": [{
"name": "Step 2",
"status": "passed"
}]
}]
}
```
And it is shown in the report like this:

The steps are nested one into another instead of being independent as was intended.
#### Workaround
To fix the report, change you code so the functions that create steps are executed synchronously:
```python
import pytest
import allure
import asyncio
@pytest.mark.asyncio
async def test_with_concurrent_steps():
fence = asyncio.Event()
async def run_with_step(name):
with allure.step(name):
await fence.wait()
async def release_fence():
fence.set()
await release_fence()
await run_with_step("Step 1")
await run_with_step("Step 2")
```
The downside is slower execution and the need to rewrite the logic in your code.
Also, see [this test](https://github.com/allure-framework/allure-python/blob/707c304677a2a5ac1b4753a2b219fb4b953b3de2/allure-pytest/test/acceptance/concurrency/async_concurrency_test.py#L7).
### Thought on possible implementation
We probably should utilize ContextVars, they were introduced for the very this issue (see [PEP567](https://peps.python.org/pep-0567/)). Cons: they require python 3.7. If we decide to move this way, we may either declare async concurrency supported in python 3.7+ only or drop python 3.6 support completely (our latest decision was to focus on 3.7+ while trying to stay 3.6-compatible, if possible).
Can\`t think of another possible way for now.
Contributor guide
Assessment
This issue has not been assessed yet.