box / box/flaky

xfail + xdist + flaky == xdist crash

Open
#127 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
398
Forks
57
PR merge metrics
No merged PRs in 30d

Description

I have a single test that has a:

```
@pytest.xfail(reason="Fails on Ubuntu, works on CentOS")
```

When that is hit in one of the test runners using xdist, the following internalerror is raised:

```
20:17:40 +0000 INTERNALERROR> Traceback (most recent call last):
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 105, in wrap_session
20:17:40 +0000 INTERNALERROR> session.exitstatus = doit(config, session) or 0
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/main.py", line 141, in _main
20:17:40 +0000 INTERNALERROR> config.hook.pytest_runtestloop(session=session)
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
20:17:40 +0000 INTERNALERROR> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
20:17:40 +0000 INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in
20:17:40 +0000 INTERNALERROR> _MultiCall(methods, kwargs, hook.spec_opts).execute()
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
20:17:40 +0000 INTERNALERROR> return _wrapped_call(hook_impl.function(*args), self.execute)
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
20:17:40 +0000 INTERNALERROR> wrap_controller.send(call_outcome)
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 280, in get_result
20:17:40 +0000 INTERNALERROR> _reraise(*ex) # noqa
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
20:17:40 +0000 INTERNALERROR> self.result = func()
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
20:17:40 +0000 INTERNALERROR> res = hook_impl.function(*args)
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 539, in pytest_runtestloop
20:17:40 +0000 INTERNALERROR> self.loop_once()
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 558, in loop_once
20:17:40 +0000 INTERNALERROR> call(**kwargs)
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 664, in slave_testreport
20:17:40 +0000 INTERNALERROR> self.sched.mark_test_complete(node, rep.item_index, rep.duration)
20:17:40 +0000 INTERNALERROR> File "/var/lib/jenkins/slave/workspace/appunit-tests/venv/local/lib/python2.7/site-packages/xdist/dsession.py", line 280, in mark_test_complete
20:17:40 +0000 INTERNALERROR> self.node2pending[node].remove(item_index)
20:17:40 +0000 INTERNALERROR> ValueError: list.remove(x): x not in list
```

This kills the entire test suite and no further tests are run. Command used to make all tests flaky:

```
pytest -v -n 6 --junitxml=junit.xml --cov --cov-report=term-missing --durations=20 --force-flaky --max-runs=3 --no-success-flaky-report
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.