pytest-dev / pytest-dev/pytest-rerunfailures

Duplicated counting test case numbers with pytest-rerunfailures plugin

Open
#76 2 comments 0 reactions 1 assignee View on GitHub

@gnikonorov is already working on this.

Since Aug 2, 2020.

Dominant language
Python
Stars
477
Forks
107
Avg merge
2d 14h
Merged PRs (30d)
21

Description

If one test cases failed in teardown method with --reruns=1, this case will be calculated as 2 cases.

Test code:

class Test1(object):

    def teardown_method(self, method):
        raise Exception('tewrdown method error')

    def test1(self):
        pass

with --reruns=1:

$ py.test test_rerun.py -v -s --junitxml=test-result.xml --reruns=1
=============================================================== test session starts ================================================================
platform darwin -- Python 2.7.14, pytest-2.9.1, py-1.7.0, pluggy-0.3.1 -- /Users/jzhang/miniconda2/envs/p2/bin/python
cachedir: .cache
rootdir: /Users/jzhang/Desktop/test_log, inifile:
plugins: rerunfailures-4.1
collected 1 items

test_rerun.py::Test1::test1 PASSED
test_rerun.py::Test1::test1 RERUN
test_rerun.py::Test1::test1 PASSED
test_rerun.py::Test1::test1 ERROR

---------------------------------------- generated xml file: /Users/jzhang/Desktop/test_log/test-result.xml ----------------------------------------
====================================================================== ERRORS ======================================================================
_________________________________________________________ ERROR at teardown of Test1.test1 _________________________________________________________

self = <test_rerun.Test1 object at 0x104073190>, method = <bound method Test1.test1 of <test_rerun.Test1 object at 0x104073190>>

    def teardown_method(self, method):
>       raise Exception('tewrdown method error')
E       Exception: tewrdown method error

test_rerun.py:4: Exception
==================================================== 2 passed, 1 error, 1 rerun in 0.02 seconds ====================================================
(p2) # ~/Desktop/test_log [18:10:33] C:1
$ more test-result.xml
<?xml version="1.0" encoding="utf-8"?><testsuite errors="1" failures="0" name="pytest" skips="0" tests="2" time="0.015"><testcase classname="test_rerun.Test1" file="test_rerun.py" line="5" name="test1" time="0.00033712387085"></testcase><testcase classname="test_rerun.Test1" file="test_rerun.py" line="5" name="test1" time="0.000302791595459"><error message="test setup failure">self = &lt;test_rerun.Test1 object at 0x104073190&gt;, method = &lt;bound method Test1.test1 of &lt;test_rerun.Test1 object at 0x104073190&gt;&gt;

    def teardown_method(self, method):
&gt;       raise Exception(&apos;tewrdown method error&apos;)
E       Exception: tewrdown method error

test_rerun.py:4: Exception</error></testcase></testsuite>

It calculated the number of test cases as 2, we can see tests="2".

when run this case without --retrun=1:

$ py.test test_rerun.py -v -s --junitxml=test-result.xml
=============================================================== test session starts ================================================================
platform darwin -- Python 2.7.14, pytest-2.9.1, py-1.7.0, pluggy-0.3.1 -- /Users/jzhang/miniconda2/envs/p2/bin/python
cachedir: .cache
rootdir: /Users/jzhang/Desktop/test_log, inifile:
plugins: rerunfailures-4.1
collected 1 items

test_rerun.py::Test1::test1 PASSED
test_rerun.py::Test1::test1 ERROR

---------------------------------------- generated xml file: /Users/jzhang/Desktop/test_log/test-result.xml ----------------------------------------
====================================================================== ERRORS ======================================================================
_________________________________________________________ ERROR at teardown of Test1.test1 _________________________________________________________

self = <test_rerun.Test1 object at 0x103d40d50>, method = <bound method Test1.test1 of <test_rerun.Test1 object at 0x103d40d50>>

    def teardown_method(self, method):
>       raise Exception('tewrdown method error')
E       Exception: tewrdown method error

test_rerun.py:4: Exception
======================================================== 1 passed, 1 error in 0.02 seconds =========================================================
(p2) # ~/Desktop/test_log [18:19:55] C:1
$ more test-result.xml
<?xml version="1.0" encoding="utf-8"?><testsuite errors="1" failures="0" name="pytest" skips="0" tests="1" time="0.015"><testcase classname="test_rerun.Test1" file="test_rerun.py" line="5" name="test1" time="0.000448226928711"><error message="test setup failure">self = &lt;test_rerun.Test1 object at 0x103d40d50&gt;, method = &lt;bound method Test1.test1 of &lt;test_rerun.Test1 object at 0x103d40d50&gt;&gt;

    def teardown_method(self, method):
&gt;       raise Exception(&apos;tewrdown method error&apos;)
E       Exception: tewrdown method error

test_rerun.py:4: Exception</error></testcase></testsuite>

It calculated the number of test cases as 1, we can see tests="1", it looks more reasonable.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.