pytest-dev / pytest-dev/pytest-html

Wrong assumption in `_post_process_reports` discards output to terminal

Open
#490 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
779
Forks
260
PR merge metrics
No merged PRs in 30d

Description

When running with --html, the processing removes printing the fixtures teardown log to the terminal.

It comes from a wrong assumption that the last report status is valid for all.
https://github.com/pytest-dev/pytest-html/blob/fdcd5c6331861c338e12537313b395300007f42e/src/pytest_html/html_report.py#L279-L282
And, as the plugin modifies the test_report object it has side-effects on other users like _pytest.terminal.py.
https://github.com/pytest-dev/pytest-html/blob/fdcd5c6331861c338e12537313b395300007f42e/src/pytest_html/html_report.py#L308-L310

Testing

Example test file:

import pytest


@pytest.fixture
def config():
    print('SETUP')
    yield
    print('TEARDOWN')


def test_example(config):
    print('CALL')
    assert 0
Expected output

In a normal execution, I would get the output:

python3 -m pytest --tb=short test_file.py 
======================== test session starts ========================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/gha/tmp/test_pytest_html
plugins: html-3.1.1, metadata-1.11.0
collected 1 item                                                    

test_file.py F                                                [100%]

============================= FAILURES ==============================
___________________________ test_example ____________________________
test_file.py:13: in test_example
    assert 0
E   assert 0
----------------------- Captured stdout setup -----------------------
SETUP
----------------------- Captured stdout call ------------------------
CALL
--------------------- Captured stdout teardown ----------------------
TEARDOWN
====================== short test summary info ======================
FAILED test_file.py::test_example - assert 0
========================= 1 failed in 0.01s =========================
Real output

When running with --html it would remove the TEARDOWN output.

python3 -m pytest --tb=short --html=/tmp/out.html test_file.py 
======================== test session starts ========================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/gha/tmp/test_pytest_html
plugins: html-3.1.1, metadata-1.11.0
collected 1 item                                                    

test_file.py F                                                [100%]

============================= FAILURES ==============================
___________________________ test_example ____________________________
test_file.py:13: in test_example
    assert 0
E   assert 0
----------------------- Captured stdout setup -----------------------
SETUP
----------------------- Captured stdout call ------------------------
CALL
------------- generated html file: file:///tmp/out.html -------------
====================== short test summary info ======================
FAILED test_file.py::test_example - assert 0
========================= 1 failed in 0.02s =========================
--- out	2022-01-19 18:04:21.025161810 +0100
+++ out_html	2022-01-19 18:04:15.325074196 +0100
@@ -15,8 +15,7 @@
 SETUP
 ----------------------------- Captured stdout call -----------------------------
 CALL
---------------------------- Captured stdout teardown ---------------------------
-TEARDOWN
+------------------ generated html file: file:///tmp/out.html -------------------
 =========================== short test summary info ============================
 FAILED test_file.py::test_example - assert 0
-============================== 1 failed in 0.01s ===============================
+============================== 1 failed in 0.02s ===============================
Debugging

By adding a some print in the plugin.py file (still on the version from pypi locally).

    def _post_process_reports(self):
        for test_name, test_reports in self.reports.items():
            import pprint
            print()
            pprint.pprint(test_reports)

It would show that, at least, "outcome" should not be taken from the last one.

python3 -m pytest --tb=short --html=/tmp/out.html test_file.py  
=========================== test session starts ============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/gha/tmp/test_pytest_html
plugins: html-3.1.1, metadata-1.11.0
collected 1 item                                                           

test_file.py F                                                       [100%]
[<TestReport 'test_file.py::test_example' when='setup' outcome='passed'>,
 <TestReport 'test_file.py::test_example' when='call' outcome='failed'>,
 <TestReport 'test_file.py::test_example' when='teardown' outcome='passed'>]


================================= FAILURES =================================
_______________________________ test_example _______________________________
test_file.py:13: in test_example
    assert 0
E   assert 0
-------------------------- Captured stdout setup ---------------------------
SETUP
--------------------------- Captured stdout call ---------------------------
CALL
---------------- generated html file: file:///tmp/out.html -----------------
========================= short test summary info ==========================
FAILED test_file.py::test_example - assert 0
============================ 1 failed in 0.02s =============================
Possible fix

When removing the overwriting on test_report.outcome and test_report.when I would get again TEARDOWN in the output.
Not checked the html output with this.

pytest-html could create a new combined test_report object instead of modifying the one given by pytest to be sure that even if this is needed, it would not change the rest of the world behavior.

I tried adding a copy before modifying test_report and it fixes the terminal output.
https://github.com/pytest-dev/pytest-html/blob/fdcd5c6331861c338e12537313b395300007f42e/src/pytest_html/html_report.py#L305-L309

             # outcome on the right comes from the outcome of the various
             #  test_reports that make up this test case
             #    we are just carrying it over to the final report.
+            import copy
+            test_report = copy.copy(test_report)
             test_report.outcome = report_outcome
             test_report.when = "call"

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at src/pytest_html/html_report.py, especially the _post_process_reports logic around the linked lines, and reproduce the issue with the provided fixture and test_example case using pytest --tb=short --html=.... Confirm that terminal output retains Captured stdout teardown while the HTML report still works without changing pytest's original report object for other consumers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.