pytest-dev / pytest-dev/pytest-xdist
xfail mark ignored if xdist slave process crashes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
When a test causes an xdist slave-worker process to crash, the test always reports a failure, even if that test is has the xfail mark. It seems preferable to keep track of xfail marks in the master, and report an xfail rather than a FAIL for a test that crashed a slave.
Obviously it's better if we can write our tests to avoid crashing the slave process, but that's not always possible. Since xdist ignores the xfail mark on crashed tests, there's no way to communicate that such a test is allowed to fail. It also breaks symmetry with the XPASS and xfail statuses reported when such tests don't crash.
Sample tests in test_foo.py:
import os
import pytest
@pytest.mark.xfail
def test_bar():
assert False
@pytest.mark.xfail
def test_baz():
os.execv('/bin/sh', ['sh', '-c', 'exit 1'])
Sample run output:
$ pytest -v -n1
========================================= test session starts ==========================================
platform linux -- Python 3.6.3, pytest-3.3.1, py-1.5.2, pluggy-0.6.0 -- /home/nick/.local/share/virtualenvs/xdist-crash-x3F5dNZR/bin/python3.6m
cachedir: .cache
rootdir: /home/nick/stuff/xdist-crash, inifile:
plugins: xdist-1.20.1, forked-0.2
[gw0] linux Python 3.6.3 cwd: /home/nick/stuff/xdist-crash
[gw0] Python 3.6.3 (default, Oct 6 2017, 08:44:35) -- [GCC 5.4.0 20160609]
gw0 [2]
scheduling tests via LoadScheduling
test_foo.py::test_bar
[gw0] [ 50%] xfail test_foo.py::test_bar
test_foo.py::test_baz
[gw0] node down: Not properly terminated
[gw0] [100%] FAILED test_foo.py::test_baz
Replacing crashed slave gw0
[gw1] linux Python 3.6.3 cwd: /home/nick/stuff/xdist-crash
[gw1] Python 3.6.3 (default, Oct 6 2017, 08:44:35) -- [GCC 5.4.0 20160609]
gw1 ok
=============================================== FAILURES ===============================================
_____________________________________________ test_foo.py ______________________________________________
[gw0] linux -- Python 3.6.3 /home/nick/.local/share/virtualenvs/xdist-crash-x3F5dNZR/bin/python3.6m
Slave 'gw0' crashed while running 'test_foo.py::test_baz'
================================= 1 failed, 1 xfailed in 1.27 seconds ==================================
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the crash with the sample tests in test_foo.py using pytest -v -n1, and compare the xfail and crashed-test output. Trace how the xdist master tracks the test mark after the slave crashes; done means the crashed test is reported as xfail rather than FAILED while preserving the existing failure summary behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100