allure-framework / allure-framework/allure-python

Report for skip is a child for unexecuted fixtures.

Open
#791 0 comments 0 reactions 0 assignees View on GitHub
bug theme:pytest
Dominant language
Python
Stars
814
Forks
260
PR merge metrics
No merged PRs in 30d

Description

[//]: # (
. Note: for support questions, please use Stackoverflow or Gitter**.
. This repository's issues are reserved for feature requests and bug reports.
.
. In case of any problems with Allure Jenkins plugin** please use the following repository
. to create an issue: https://github.com/jenkinsci/allure-plugin/issues
.
. Make sure you have a clear name for your issue. The name should start with a capital
. letter and no dot is required in the end of the sentence. An example of good issue names:
.
. - The report is broken in IE11
. - Add an ability to disable default plugins
. - Support emoji in test descriptions
)

#### I'm submitting a ...
- [x] bug report
- [ ] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.

#### What is the current behavior?

Skip Tests are reported like they had actions/steps.

#### If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

Generate report for with `-s`:

```python
import pytest
import allure

@pytest.fixture(scope='module', autouse=True)
def module_dep(request: pytest.FixtureRequest):
with allure.step(f'module start {request.node.nodeid}'):
pass
print('<<< start >>>')
yield
with allure.step(f'module end {request.node.nodeid}'):
pass
print('<<>>')

@pytest.fixture(autouse=True)
def tc_dep(request: pytest.FixtureRequest):
with allure.step(f'test start {request.node.nodeid}'):
pass
print('<< start test >>')
yield
with allure.step(f'test end {request.node.nodeid}'):
pass
print('<>')

class TestFoo:

@pytest.mark.skip
def test_a(self):
pass

def test_b(self):
pass
```

Observation:
- test_a has fixtures steps that does not own
- pytest terminal reports at the time of skip - nothing was run

![image](https://github.com/allure-framework/allure-python/assets/1913863/c77b53f4-7307-4844-a42f-c52d503c9326)

I assume the bug is in `allure python` due to

```json
{
"uuid": "1adbda40-73cc-47ab-a3f5-154adeab99db",
"children": [
"d0c0a1b8-6b3e-496d-8758-e599ed9facf0",
"a16e7ffe-92f7-4212-ab95-6dc6486661ab"
],
"befores": [
{
"name": "tc_dep",
"status": "passed",
"steps": [
{
"name": "test start test_session_skip.py::TestFoo::test_b",
"status": "passed"
}
]
}
],
"afters": [
{
"name": "tc_dep::0",
"status": "passed",
"steps": [
{
"name": "test end test_session_skip.py::TestFoo::test_b",
"status": "passed"
}
]
}
]
}
```
where `children` are `test_a` and `test_b` - I'd expect only test_b should be among children. Sorry hardly understand allure models. (ps: start/stop removed for a focused view.)

Long thinking on it, I asume there might be reason why each test shows e.g. "module" or "session" setup/teardown fixture. Would be nice if it can be visually clear "shared" parents.

#### What is the expected behavior?

Nodes/containers do not have nodes that do not belong.

#### What is the motivation / use case for changing the behavior?

Bug. This is very confusing. And making think that **unconditionally skipped test somehow executed fixtures**

I spent couple hours in prod environment checking "why do I see it", while provided steps above - explained anomaly.

For a heavy setup/tear down, visual feedback for "children" a bit confusing.

_A bit offtopic..._

```
pytest execution protocol:
- module fixture - setup
- class fixture - setup
- test fixture - setup
- test_foo
- test fixture - teardown
- test fixture - setup
- test_baz
- test fixture - teardown
- class fixture - teardown
- module fixture - teardown
```
in allure report shared "class", "module", "package", "session" are shown with each test... and it seems from this issue "function" sometimes too. Nice to have visual indication of shared nodes. _Glamour-allure tries to do it ([here](https://github.com/Denis-Alexeev/pytest-glamor-allure/raw/master/assets/scope_before.png))_ but I cannot aford more dependencies and nice if it can be considered "from the box". I'd really like to try to understand is this `Item` was the first and started "higher-scope" fixtures, or it is just "reported here" (a pointer :) ).

#### Please tell us about your environment:

- Allure version: 2.26 or 2.27 (tried both from https://github.com/TobiX/allure-cli via sh script `docker run --rm -v $PWD:/work -it tobix/allure-cli $@`)
- Test framework: pytest@7.4.4 or pytest@8.0
- Allure adaptor: allure-pytest@2.13.2

#### Other information

[//]: # (
. e.g. detailed explanation, stacktraces, related issues, suggestions
. how to fix, links for us to have more context, eg. Stackoverflow, Gitter etc
)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.