pytest-dev / pytest-dev/pytest-xdist

xfail mark ignored if xdist slave process crashes

Open
#276 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

When a test causes an xdist slave-worker process to crash, the test always reports a failure, even if that test is has the xfail mark. It seems preferable to keep track of xfail marks in the master, and report an xfail rather than a FAIL for a test that crashed a slave.

Obviously it's better if we can write our tests to avoid crashing the slave process, but that's not always possible. Since xdist ignores the xfail mark on crashed tests, there's no way to communicate that such a test is allowed to fail. It also breaks symmetry with the XPASS and xfail statuses reported when such tests don't crash.

Sample tests in test_foo.py:

import os
import pytest

@pytest.mark.xfail
def test_bar():
    assert False

@pytest.mark.xfail
def test_baz():
    os.execv('/bin/sh', ['sh', '-c', 'exit 1'])

Sample run output:

$ pytest -v -n1
========================================= test session starts ==========================================
platform linux -- Python 3.6.3, pytest-3.3.1, py-1.5.2, pluggy-0.6.0 -- /home/nick/.local/share/virtualenvs/xdist-crash-x3F5dNZR/bin/python3.6m
cachedir: .cache
rootdir: /home/nick/stuff/xdist-crash, inifile:
plugins: xdist-1.20.1, forked-0.2
[gw0] linux Python 3.6.3 cwd: /home/nick/stuff/xdist-crash
[gw0] Python 3.6.3 (default, Oct  6 2017, 08:44:35)  -- [GCC 5.4.0 20160609]
gw0 [2]
scheduling tests via LoadScheduling

test_foo.py::test_bar
[gw0] [ 50%] xfail test_foo.py::test_bar
test_foo.py::test_baz
[gw0] node down: Not properly terminated
[gw0] [100%] FAILED test_foo.py::test_baz
Replacing crashed slave gw0
[gw1] linux Python 3.6.3 cwd: /home/nick/stuff/xdist-crash
[gw1] Python 3.6.3 (default, Oct  6 2017, 08:44:35)  -- [GCC 5.4.0 20160609]
gw1 ok
=============================================== FAILURES ===============================================
_____________________________________________ test_foo.py ______________________________________________
[gw0] linux -- Python 3.6.3 /home/nick/.local/share/virtualenvs/xdist-crash-x3F5dNZR/bin/python3.6m
Slave 'gw0' crashed while running 'test_foo.py::test_baz'
================================= 1 failed, 1 xfailed in 1.27 seconds ==================================

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the crash with the sample tests in test_foo.py using pytest -v -n1, and compare the xfail and crashed-test output. Trace how the xdist master tracks the test mark after the slave crashes; done means the crashed test is reported as xfail rather than FAILED while preserving the existing failure summary behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.