pytest-dev / pytest-dev/pytest-html
Wrong assumption in `_post_process_reports` discards output to terminal
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 779
- Forks
- 260
- PR merge metrics
- No merged PRs in 30d
Description
When running with --html, the processing removes printing the fixtures teardown log to the terminal.
It comes from a wrong assumption that the last report status is valid for all.
https://github.com/pytest-dev/pytest-html/blob/fdcd5c6331861c338e12537313b395300007f42e/src/pytest_html/html_report.py#L279-L282
And, as the plugin modifies the test_report object it has side-effects on other users like _pytest.terminal.py.
https://github.com/pytest-dev/pytest-html/blob/fdcd5c6331861c338e12537313b395300007f42e/src/pytest_html/html_report.py#L308-L310
Testing
Example test file:
import pytest
@pytest.fixture
def config():
print('SETUP')
yield
print('TEARDOWN')
def test_example(config):
print('CALL')
assert 0
Expected output
In a normal execution, I would get the output:
python3 -m pytest --tb=short test_file.py
======================== test session starts ========================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/gha/tmp/test_pytest_html
plugins: html-3.1.1, metadata-1.11.0
collected 1 item
test_file.py F [100%]
============================= FAILURES ==============================
___________________________ test_example ____________________________
test_file.py:13: in test_example
assert 0
E assert 0
----------------------- Captured stdout setup -----------------------
SETUP
----------------------- Captured stdout call ------------------------
CALL
--------------------- Captured stdout teardown ----------------------
TEARDOWN
====================== short test summary info ======================
FAILED test_file.py::test_example - assert 0
========================= 1 failed in 0.01s =========================
Real output
When running with --html it would remove the TEARDOWN output.
python3 -m pytest --tb=short --html=/tmp/out.html test_file.py
======================== test session starts ========================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/gha/tmp/test_pytest_html
plugins: html-3.1.1, metadata-1.11.0
collected 1 item
test_file.py F [100%]
============================= FAILURES ==============================
___________________________ test_example ____________________________
test_file.py:13: in test_example
assert 0
E assert 0
----------------------- Captured stdout setup -----------------------
SETUP
----------------------- Captured stdout call ------------------------
CALL
------------- generated html file: file:///tmp/out.html -------------
====================== short test summary info ======================
FAILED test_file.py::test_example - assert 0
========================= 1 failed in 0.02s =========================
--- out 2022-01-19 18:04:21.025161810 +0100
+++ out_html 2022-01-19 18:04:15.325074196 +0100
@@ -15,8 +15,7 @@
SETUP
----------------------------- Captured stdout call -----------------------------
CALL
---------------------------- Captured stdout teardown ---------------------------
-TEARDOWN
+------------------ generated html file: file:///tmp/out.html -------------------
=========================== short test summary info ============================
FAILED test_file.py::test_example - assert 0
-============================== 1 failed in 0.01s ===============================
+============================== 1 failed in 0.02s ===============================
Debugging
By adding a some print in the plugin.py file (still on the version from pypi locally).
def _post_process_reports(self):
for test_name, test_reports in self.reports.items():
import pprint
print()
pprint.pprint(test_reports)
It would show that, at least, "outcome" should not be taken from the last one.
python3 -m pytest --tb=short --html=/tmp/out.html test_file.py
=========================== test session starts ============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/gha/tmp/test_pytest_html
plugins: html-3.1.1, metadata-1.11.0
collected 1 item
test_file.py F [100%]
[<TestReport 'test_file.py::test_example' when='setup' outcome='passed'>,
<TestReport 'test_file.py::test_example' when='call' outcome='failed'>,
<TestReport 'test_file.py::test_example' when='teardown' outcome='passed'>]
================================= FAILURES =================================
_______________________________ test_example _______________________________
test_file.py:13: in test_example
assert 0
E assert 0
-------------------------- Captured stdout setup ---------------------------
SETUP
--------------------------- Captured stdout call ---------------------------
CALL
---------------- generated html file: file:///tmp/out.html -----------------
========================= short test summary info ==========================
FAILED test_file.py::test_example - assert 0
============================ 1 failed in 0.02s =============================
Possible fix
When removing the overwriting on test_report.outcome and test_report.when I would get again TEARDOWN in the output.
Not checked the html output with this.
pytest-html could create a new combined test_report object instead of modifying the one given by pytest to be sure that even if this is needed, it would not change the rest of the world behavior.
I tried adding a copy before modifying test_report and it fixes the terminal output.
https://github.com/pytest-dev/pytest-html/blob/fdcd5c6331861c338e12537313b395300007f42e/src/pytest_html/html_report.py#L305-L309
# outcome on the right comes from the outcome of the various
# test_reports that make up this test case
# we are just carrying it over to the final report.
+ import copy
+ test_report = copy.copy(test_report)
test_report.outcome = report_outcome
test_report.when = "call"
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at src/pytest_html/html_report.py, especially the _post_process_reports logic around the linked lines, and reproduce the issue with the provided fixture and test_example case using pytest --tb=short --html=.... Confirm that terminal output retains Captured stdout teardown while the HTML report still works without changing pytest's original report object for other consumers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100