Shopify / Shopify/ci-queue

pytest: TestCase failures reported as passing

Open
#91 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
239
Forks
36
Avg merge
28m
Merged PRs (30d)
3

Description

With this test suite:

# test_example.py
from unittest import TestCase

class TestExample(TestCase):
    def test_fail(self):
        assert False

Running works as expected:

$ pytest -p ciqueue.pytest --queue 'redis://localhost:6379/0?worker=0&build=repro&retry=0&timeout=10' test_example.py
========================================================= test session starts =========================================================
platform darwin -- Python 2.7.13, pytest-4.0.1, py-1.7.0, pluggy-0.6.0
...
collected 1 item

test_example.py F

============================================================== FAILURES ===============================================================
________________________________________________________ TestExample.test_fail ________________________________________________________

self = <test_example.TestExample testMethod=test_fail>

    def test_fail(self):
>       assert False
E       AssertionError: assert False

test_example.py:5: AssertionError
====================================================== 1 failed in 0.06 seconds =======================================================

However, reporting shows that the test passed:

$ pytest -p ciqueue.pytest_report --queue 'redis://localhost:6379/0?worker=0&build=repro&retry=0&timeout=10' test_example.py
========================================================= test session starts =========================================================
platform darwin -- Python 2.7.13, pytest-4.0.1, py-1.7.0, pluggy-0.6.0
...
collected 1 item

test_example.py .                                                                                                               [100%]

====================================================== 1 passed in 0.02 seconds =======================================================

This appears to be due to ciqueue's makereport triggering before the one in pytest which is responsible for setting call.excinfo for TestCase-style tests.

From these docs it looks like:

  1. the ordering of multiple tryfirst=True hooks is undefined
  2. hookwrapper=True can execute code before a tryfirst hook

So, one fix could be to get pytest to switch to using hookwrapper. I figured I'd raise it with you folks first, though, in case it's something that's easier to fix on this end (by making ciqueue execute later, somehow).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the shown TestCase example and the ciqueue.pytest_report plugin command. Read the pytest unittest hook referenced in the issue and the ciqueue reporting hook to verify their ordering and handling of call.excinfo. Done means a failing TestCase is reported as failed rather than passed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.