xdist continues test execution in the background with '--exitfirst'

When running pytest with --exitfirst or --maxfail=<some non zero value>, the main pytest session exits on the first failure as expected. However, the test workers are still executing the tests in the background.

The --exitfirst flag is very powerful when investigating test failures in scenarios where the tests have side-effects and starting the next test may destroy valuable execution traces 😃

Here is a minimal test suite to reproduce the issue:

import os
import unittest

logfile = open('/tmp/pytest.log', 'a')
worker = os.environ.get('PYTEST_XDIST_WORKER', 'single')

def log(line):
    logfile.write("[%s] %s\n" % (worker, line))

class TestPytest(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        log("setUpClass")

    def setUp(self):
        log("setUp")

    def test_1(self):
        log("test_1")

    def test_2(self):
        log("test_2")
        assert False

    def test_3(self):
        log("test_3")

Test run with xdist:

rm -f /tmp/pytest.log && pytest-3 -x -d --tx popen//python=python3 test_pytest.py; cat /tmp/pytest.log
(actual test output redacted)
[gw0] setUpClass
[gw0] setUp
[gw0] test_1
[gw0] setUp
[gw0] test_2
[gw0] setUp
[gw0] test_3

Test run without xdist:

rm -f /tmp/pytest.log && pytest-3 -x test_pytest.py; cat /tmp/pytest.log
(actual test output redacted)
[single] setUpClass
[single] setUp
[single] test_1
[single] setUp
[single] test_2

Here is a partial patch to address this issue:

diff --git a/xdist/remote.py b/xdist/remote.py
index 346d6e5..36fc577 100644
--- a/xdist/remote.py
+++ b/xdist/remote.py
@@ -85,6 +85,8 @@ class WorkerInteractor:
         duration = time.time() - start
         self.sendevent("runtest_protocol_complete", item_index=self.item_index,
                        duration=duration)
+        if self.session.shouldfail:
+            raise self.session.Failed(self.session.shouldfail)
 
     def pytest_collection_finish(self, session):
         self.sendevent(
@@ -106,6 +108,8 @@ class WorkerInteractor:
         data["worker_id"] = self.workerid
         assert self.session.items[self.item_index].nodeid == report.nodeid
         self.sendevent("testreport", data=data)
+        if report.failed and self.config.getvalue("maxfail"):
+            self.session.shouldfail = "Immediately exit worker on failure"
 
     def pytest_collectreport(self, report):
         data = serialize_report(report)

When maxfail is set, this patch causes the worker to exit on the first failure. This implements the behavior of --exitfirst in the worker. However, when using --maxfail=<some non zero/non one value>, this produces a stacktrace.

Start with xdist/remote.py and reproduce the behavior using the minimal unittest suite and the provided pytest-3 xdist commands. Trace how worker reports handle failures and maxfail, then verify that workers stop without continuing background tests and that non-one maxfail values do not produce a traceback.