pytest: TestCase failures reported as passing
Nobody has claimed this yet.
- Dominant language
- Ruby
- Stars
- 239
- Forks
- 36
- Avg merge
- 28m
- Merged PRs (30d)
- 3
Description
With this test suite:
# test_example.py
from unittest import TestCase
class TestExample(TestCase):
def test_fail(self):
assert False
Running works as expected:
$ pytest -p ciqueue.pytest --queue 'redis://localhost:6379/0?worker=0&build=repro&retry=0&timeout=10' test_example.py
========================================================= test session starts =========================================================
platform darwin -- Python 2.7.13, pytest-4.0.1, py-1.7.0, pluggy-0.6.0
...
collected 1 item
test_example.py F
============================================================== FAILURES ===============================================================
________________________________________________________ TestExample.test_fail ________________________________________________________
self = <test_example.TestExample testMethod=test_fail>
def test_fail(self):
> assert False
E AssertionError: assert False
test_example.py:5: AssertionError
====================================================== 1 failed in 0.06 seconds =======================================================
However, reporting shows that the test passed:
$ pytest -p ciqueue.pytest_report --queue 'redis://localhost:6379/0?worker=0&build=repro&retry=0&timeout=10' test_example.py
========================================================= test session starts =========================================================
platform darwin -- Python 2.7.13, pytest-4.0.1, py-1.7.0, pluggy-0.6.0
...
collected 1 item
test_example.py . [100%]
====================================================== 1 passed in 0.02 seconds =======================================================
This appears to be due to ciqueue's makereport triggering before the one in pytest which is responsible for setting call.excinfo for TestCase-style tests.
From these docs it looks like:
- the ordering of multiple
tryfirst=Truehooks is undefined hookwrapper=Truecan execute code before atryfirsthook
So, one fix could be to get pytest to switch to using hookwrapper. I figured I'd raise it with you folks first, though, in case it's something that's easier to fix on this end (by making ciqueue execute later, somehow).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with the shown TestCase example and the ciqueue.pytest_report plugin command. Read the pytest unittest hook referenced in the issue and the ciqueue reporting hook to verify their ordering and handling of call.excinfo. Done means a failing TestCase is reported as failed rather than passed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100