allure-framework / allure-framework/allure-python

Add support for coroutine-based concurrency

Open
#720 4 comments 2 reactions 0 assignees View on GitHub
task:new feature theme:core theme:pytest
Dominant language
Python
Stars
814
Forks
260
PR merge metrics
No merged PRs in 30d

Description

Currently, we don\`t support running allure-python in an async function (e.g. through asyncio event loop or a similar coroutine execution engine).

This is important, because
1. Async-based frameworks and libraries are growing in popularity (`aiohttp`, `aiofile`, `aiopg`, etc.), so we need to support testing asynchronous code
2. E2E tests are heavily IO-bound. The ability to run them concurrently may improve tests performance significantly.

#### Coroutines and contexts
A coroutine in python is created with a coroutine declaration, i.e. a function with the `async` keyword. Inside a function the `await` keyword can be used to wait for an asynchronous operation to complete. A coroutine execution engine can then pause the coroutine and resume execution of another one, increasing performance of IO-bound computations.

Without proper means a coroutine paused inside a context manager may leak unintended global state into another coroutine.

#### Basic illustration

Suppose we have the following abstract code:
```python
async def f():
with change_global_state():
await long_io_operation()

async def g():
use_global_state()
```

If we create two coroutines (one with `f()` and the other one with `g()`) and schedule both coroutines for execution, it is possible that `g` will use the state, not expected to be visible outside `f`.

Below is an example, how both coroutines could be run with `asyncio`:

```python
import asyncio

async def execute_all():
await asyncio.gather(f(), g())

asyncio.run(execute_all())
```

There are two general scenarios we can face while generating allure report. They are described below.

### Scenario 1: running multiple tests concurrently
This scenario requires a pytest plugin capable of executing async tests concurrently, like [pytest-asyncio-cooperative](https://github.com/willemt/pytest-asyncio-cooperative). Unfortunately, this particular plugin currently doesn\`t work with allure-pytest, so the remaining content in this section describes more of a theoretical case, as if the plugin actually works.

If we assume the plugin works, the test may looks like this ([also, see here](https://github.com/allure-framework/allure-python/blob/a86a8f4172681257bc64eb3ff8feeb78592090c2/allure-pytest/test/acceptance/concurrency/async_concurrency_test.py#L51)):
```python
import allure
import asyncio
import pytest

@pytest.fixture(scope="session")
async def fence():
yield asyncio.Event()

@pytest.mark.asyncio_cooperative
async def test_with_async_operation(fence):
with allure.step("Wait for a fence"):
await fence.wait()

@pytest.mark.asyncio_cooperative
async def test_without_async_operation(fence):
with allure.step("Setting a fence"):
fence.set()
```

After `test_with_async_operation` is paused, the event loop starts executing `test_without_async_operation`. The step `"Setting a fence"` is created inside the `"Wait for a fence""` step context, emitting an invalid test result: the first test result contains two nested steps instead of one, while the second one contains no steps whatsoever.

### Scenario 2: running multiple steps concurrently inside one test
This one is easily achieved either with a self-written fixture, or with [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio).

Given we have pytest-asyncio installed and the following code:
```python
import pytest
import allure
import asyncio

@pytest.mark.asyncio
async def test_with_concurrent_steps():
fence = asyncio.Event()
async def run_with_step(name):
with allure.step(name):
await fence.wait()
async def release_fence():
fence.set()
await asyncio.gather(
run_with_step("Step 1"),
run_with_step("Step 2"),
release_fence()
)
```

Note: the steps could've been executed with `asyncio`'s `create_task` function instead with the same effect.

The following `*result.json` is generated (most unrelated fields are omitted):
```json
{
"name": "test_with_concurrent_steps",
"status": "passed",
"steps": [{
"name": "Step 1",
"status": "passed",
"steps": [{
"name": "Step 2",
"status": "passed"
}]
}]
}
```

And it is shown in the report like this:
![image](https://user-images.githubusercontent.com/17935127/208509785-7a1f1e10-78cd-4bf7-aecc-b7a1e7bc6820.png)

The steps are nested one into another instead of being independent as was intended.

#### Workaround
To fix the report, change you code so the functions that create steps are executed synchronously:

```python
import pytest
import allure
import asyncio

@pytest.mark.asyncio
async def test_with_concurrent_steps():
fence = asyncio.Event()
async def run_with_step(name):
with allure.step(name):
await fence.wait()
async def release_fence():
fence.set()
await release_fence()
await run_with_step("Step 1")
await run_with_step("Step 2")
```

The downside is slower execution and the need to rewrite the logic in your code.

Also, see [this test](https://github.com/allure-framework/allure-python/blob/707c304677a2a5ac1b4753a2b219fb4b953b3de2/allure-pytest/test/acceptance/concurrency/async_concurrency_test.py#L7).

### Thought on possible implementation
We probably should utilize ContextVars, they were introduced for the very this issue (see [PEP567](https://peps.python.org/pep-0567/)). Cons: they require python 3.7. If we decide to move this way, we may either declare async concurrency supported in python 3.7+ only or drop python 3.6 support completely (our latest decision was to focus on 3.7+ while trying to stay 3.6-compatible, if possible).

Can\`t think of another possible way for now.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.