allure-framework / allure-framework/allure-python

Report for skip is a child for unexecuted fixtures.

Offen
#791 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug theme:pytest
Vorherrschende Sprache
Python
Sterne
814
Forks
260
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

[//]: # (
. Note: for support questions, please use Stackoverflow or Gitter**.
. This repository's issues are reserved for feature requests and bug reports.
.
. In case of any problems with Allure Jenkins plugin** please use the following repository
. to create an issue: https://github.com/jenkinsci/allure-plugin/issues
.
. Make sure you have a clear name for your issue. The name should start with a capital
. letter and no dot is required in the end of the sentence. An example of good issue names:
.
. - The report is broken in IE11
. - Add an ability to disable default plugins
. - Support emoji in test descriptions
)

#### I'm submitting a ...
- [x] bug report
- [ ] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.

#### What is the current behavior?

Skip Tests are reported like they had actions/steps.

#### If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

Generate report for with `-s`:

```python
import pytest
import allure

@pytest.fixture(scope='module', autouse=True)
def module_dep(request: pytest.FixtureRequest):
with allure.step(f'module start {request.node.nodeid}'):
pass
print('<<< start >>>')
yield
with allure.step(f'module end {request.node.nodeid}'):
pass
print('<<>>')

@pytest.fixture(autouse=True)
def tc_dep(request: pytest.FixtureRequest):
with allure.step(f'test start {request.node.nodeid}'):
pass
print('<< start test >>')
yield
with allure.step(f'test end {request.node.nodeid}'):
pass
print('<>')

class TestFoo:

@pytest.mark.skip
def test_a(self):
pass

def test_b(self):
pass
```

Observation:
- test_a has fixtures steps that does not own
- pytest terminal reports at the time of skip - nothing was run

![image](https://github.com/allure-framework/allure-python/assets/1913863/c77b53f4-7307-4844-a42f-c52d503c9326)

I assume the bug is in `allure python` due to

```json
{
"uuid": "1adbda40-73cc-47ab-a3f5-154adeab99db",
"children": [
"d0c0a1b8-6b3e-496d-8758-e599ed9facf0",
"a16e7ffe-92f7-4212-ab95-6dc6486661ab"
],
"befores": [
{
"name": "tc_dep",
"status": "passed",
"steps": [
{
"name": "test start test_session_skip.py::TestFoo::test_b",
"status": "passed"
}
]
}
],
"afters": [
{
"name": "tc_dep::0",
"status": "passed",
"steps": [
{
"name": "test end test_session_skip.py::TestFoo::test_b",
"status": "passed"
}
]
}
]
}
```
where `children` are `test_a` and `test_b` - I'd expect only test_b should be among children. Sorry hardly understand allure models. (ps: start/stop removed for a focused view.)

Long thinking on it, I asume there might be reason why each test shows e.g. "module" or "session" setup/teardown fixture. Would be nice if it can be visually clear "shared" parents.

#### What is the expected behavior?

Nodes/containers do not have nodes that do not belong.

#### What is the motivation / use case for changing the behavior?

Bug. This is very confusing. And making think that **unconditionally skipped test somehow executed fixtures**

I spent couple hours in prod environment checking "why do I see it", while provided steps above - explained anomaly.

For a heavy setup/tear down, visual feedback for "children" a bit confusing.

_A bit offtopic..._

```
pytest execution protocol:
- module fixture - setup
- class fixture - setup
- test fixture - setup
- test_foo
- test fixture - teardown
- test fixture - setup
- test_baz
- test fixture - teardown
- class fixture - teardown
- module fixture - teardown
```
in allure report shared "class", "module", "package", "session" are shown with each test... and it seems from this issue "function" sometimes too. Nice to have visual indication of shared nodes. _Glamour-allure tries to do it ([here](https://github.com/Denis-Alexeev/pytest-glamor-allure/raw/master/assets/scope_before.png))_ but I cannot aford more dependencies and nice if it can be considered "from the box". I'd really like to try to understand is this `Item` was the first and started "higher-scope" fixtures, or it is just "reported here" (a pointer :) ).

#### Please tell us about your environment:

- Allure version: 2.26 or 2.27 (tried both from https://github.com/TobiX/allure-cli via sh script `docker run --rm -v $PWD:/work -it tobix/allure-cli $@`)
- Test framework: pytest@7.4.4 or pytest@8.0
- Allure adaptor: allure-pytest@2.13.2

#### Other information

[//]: # (
. e.g. detailed explanation, stacktraces, related issues, suggestions
. how to fix, links for us to have more context, eg. Stackoverflow, Gitter etc
)

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne damit, das minimale pytest-Beispiel mit allure-pytest 2.13.2 auszuführen und den generierten Container-JSON zu untersuchen, insbesondere die children für den übersprungenen Test und seine fixtures. Verfolge, wie übersprungene Tests sowie das Setup und Teardown von fixtures gemeldet werden. Erledigt bedeutet, dass ein unbedingt übersprungener Test nicht als child von fixture-Containern enthalten ist und ein Regressionstest das Beispiel abdeckt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
testing
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
38/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.