pytest-xdist escapes from PyPy virtualenv

This is not a small reproducible example, but I still hope someone may find the report informative.

Some context

I have been running the scikit-learn test suite under PyPy3-6.0. Generally it passes except for one segfault.

• if the test that segfaults is skipped, all tests pass (cf log in https://circleci.com/gh/scikit-learn/scikit-learn/33564)
• if the full test suite is run using pytest-xdist with -n 1 to recover after this 1 segfault, other unrelated tests start to fail (cf log in https://circleci.com/gh/scikit-learn/scikit-learn/33546). The order in which tests are run does not matter, there are no setup step, so there is no logical reason why these additional tests would fail.

More interestingly is that

• the traceback (see e.g. https://github.com/scikit-learn/scikit-learn/pull/12039#issuecomment-422024504) shows first lines in the PyPy virtualenv, but ends up somewhere in the standard library of the system installed CPython where it should not have access,

pypy-env/site-packages/numpy/distutils/system_info.py:1459: in get_atlas_version
    c = cmd_config(Distribution())
[...]
        if not isinstance(dist, Distribution):
>           raise TypeError("dist must be a Distribution instance")
E           TypeError: dist must be a Distribution instance

/usr/local/lib-python/3/distutils/cmd.py:57: TypeError

• tests marked with pytest.mark.skipif(platform.python_implementation() == 'PyPy') are run even though they shouldn't. They are indeed not run without pytest-xdist

Both of these points make me believe that while we start running tests with the PyPy interpreter, after the segfault in one of the tests, the new process is somehow recreated using the CPython interpreter instead.

Do you think this is plausible?

Another possibility could be some issue in the interaction between pytest-xdist and joblib.Parallel / loky.

Start by reproducing the reported PyPy3 failure with pytest-xdist using -n 1, comparing the linked CircleCI and pull-request logs with a run without xdist. Investigate whether the post-segfault process uses system CPython and whether PyPy-specific skipif tests run; done means the interpreter and skip behavior remain correct after recovery.