allure-framework / allure-framework/allure-python

Report for skip is a child for unexecuted fixtures.

Aperta
#791 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
bug theme:pytest
Lingua principale
Python
Stelle
814
Fork
260
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

[//]: # (
. Note: for support questions, please use Stackoverflow or Gitter**.
. This repository's issues are reserved for feature requests and bug reports.
.
. In case of any problems with Allure Jenkins plugin** please use the following repository
. to create an issue: https://github.com/jenkinsci/allure-plugin/issues
.
. Make sure you have a clear name for your issue. The name should start with a capital
. letter and no dot is required in the end of the sentence. An example of good issue names:
.
. - The report is broken in IE11
. - Add an ability to disable default plugins
. - Support emoji in test descriptions
)

#### I'm submitting a ...
- [x] bug report
- [ ] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.

#### What is the current behavior?

Skip Tests are reported like they had actions/steps.

#### If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

Generate report for with `-s`:

```python
import pytest
import allure

@pytest.fixture(scope='module', autouse=True)
def module_dep(request: pytest.FixtureRequest):
with allure.step(f'module start {request.node.nodeid}'):
pass
print('<<< start >>>')
yield
with allure.step(f'module end {request.node.nodeid}'):
pass
print('<<>>')

@pytest.fixture(autouse=True)
def tc_dep(request: pytest.FixtureRequest):
with allure.step(f'test start {request.node.nodeid}'):
pass
print('<< start test >>')
yield
with allure.step(f'test end {request.node.nodeid}'):
pass
print('<>')

class TestFoo:

@pytest.mark.skip
def test_a(self):
pass

def test_b(self):
pass
```

Observation:
- test_a has fixtures steps that does not own
- pytest terminal reports at the time of skip - nothing was run

![image](https://github.com/allure-framework/allure-python/assets/1913863/c77b53f4-7307-4844-a42f-c52d503c9326)

I assume the bug is in `allure python` due to

```json
{
"uuid": "1adbda40-73cc-47ab-a3f5-154adeab99db",
"children": [
"d0c0a1b8-6b3e-496d-8758-e599ed9facf0",
"a16e7ffe-92f7-4212-ab95-6dc6486661ab"
],
"befores": [
{
"name": "tc_dep",
"status": "passed",
"steps": [
{
"name": "test start test_session_skip.py::TestFoo::test_b",
"status": "passed"
}
]
}
],
"afters": [
{
"name": "tc_dep::0",
"status": "passed",
"steps": [
{
"name": "test end test_session_skip.py::TestFoo::test_b",
"status": "passed"
}
]
}
]
}
```
where `children` are `test_a` and `test_b` - I'd expect only test_b should be among children. Sorry hardly understand allure models. (ps: start/stop removed for a focused view.)

Long thinking on it, I asume there might be reason why each test shows e.g. "module" or "session" setup/teardown fixture. Would be nice if it can be visually clear "shared" parents.

#### What is the expected behavior?

Nodes/containers do not have nodes that do not belong.

#### What is the motivation / use case for changing the behavior?

Bug. This is very confusing. And making think that **unconditionally skipped test somehow executed fixtures**

I spent couple hours in prod environment checking "why do I see it", while provided steps above - explained anomaly.

For a heavy setup/tear down, visual feedback for "children" a bit confusing.

_A bit offtopic..._

```
pytest execution protocol:
- module fixture - setup
- class fixture - setup
- test fixture - setup
- test_foo
- test fixture - teardown
- test fixture - setup
- test_baz
- test fixture - teardown
- class fixture - teardown
- module fixture - teardown
```
in allure report shared "class", "module", "package", "session" are shown with each test... and it seems from this issue "function" sometimes too. Nice to have visual indication of shared nodes. _Glamour-allure tries to do it ([here](https://github.com/Denis-Alexeev/pytest-glamor-allure/raw/master/assets/scope_before.png))_ but I cannot aford more dependencies and nice if it can be considered "from the box". I'd really like to try to understand is this `Item` was the first and started "higher-scope" fixtures, or it is just "reported here" (a pointer :) ).

#### Please tell us about your environment:

- Allure version: 2.26 or 2.27 (tried both from https://github.com/TobiX/allure-cli via sh script `docker run --rm -v $PWD:/work -it tobix/allure-cli $@`)
- Test framework: pytest@7.4.4 or pytest@8.0
- Allure adaptor: allure-pytest@2.13.2

#### Other information

[//]: # (
. e.g. detailed explanation, stacktraces, related issues, suggestions
. how to fix, links for us to have more context, eg. Stackoverflow, Gitter etc
)

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia eseguendo l’esempio minimo di pytest con allure-pytest 2.13.2 e ispeziona il JSON del container generato, in particolare i children del test saltato e delle sue fixtures. Traccia il modo in cui vengono riportati i test saltati e il setup/teardown delle fixtures. Il lavoro è completo quando un test saltato incondizionatamente non è incluso come child dei container delle fixtures e un test di regressione copre l’esempio.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
testing
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
38/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.