Running tests fails with "node down: Not properly terminated", maybe execnet related?

We recently moved to Python 3.4 and when running our tests with pytest-xdist installed and in parallel with (-n) sometimes the tests fail with the below error. The traceback is from using faulthandler. The tests consistently fail during tests that use celery tasks and that have the format:

@override_settings(task_always_eager=True)
def test_example(self):

Packages used:
execnet-1.5.0
pytest-3.0.6
xdist-1.15.0

Here is the error log:

platform linux -- Python 3.4.5, pytest-3.0.6, py-1.4.32, pluggy-0.4.0
Django settings: gameserver.settings.test (from environment variable)
rootdir: /home/jcazacu/Repos/main_repo/ares-game-server, inifile: tox.ini
plugins: xdist-1.15.0, faulthandler-1.4.1, django-3.1.2, cov-2.5.1, celery-4.0.2
gw0 [535] / gw1 [535] / gw2 [535] / gw3 [535]
scheduling tests via LoadScheduling
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Fatal Python error: Segmentation fault

Thread 0x00007fbd16d3d700 (most recent call first):
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 386 in read
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 418 in from_io
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 954 in _thread_receiver
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 213 in run
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn

Current thread 0x00007fbd1eed2740 (most recent call first):
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/backends/sqlite3/base.py", line 335 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/backends/utils.py", line 62 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/backends/base/base.py", line 288 in _savepoint_rollback
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/backends/base/base.py", line 328 in savepoint_rollback
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/db/transaction.py", line 243 in exit
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/test/testcases.py", line 1004 in _rollback_atomics
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/test/testcases.py", line 1066 in _fixture_teardown
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/test/testcases.py", line 908 in _post_teardown
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/channels/tests/base.py", line 57 in _post_teardown
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/django/test/testcases.py", line 216 in call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/unittest.py", line 157 in runtest
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 104 in pytest_runtest_call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in init
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 151 in
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 163 in init
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 151 in call_runtest_hook
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 133 in call_and_report
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 79 in runtestprotocol
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/runner.py", line 66 in pytest_runtest_protocol
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in init
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in init
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in call
File "", line 77 in run_tests
File "", line 61 in pytest_runtestloop
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in call
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/main.py", line 133 in _main
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/main.py", line 98 in wrap_session
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/main.py", line 127 in pytest_cmdline_main
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in call
File "", line 159 in
File "", line 1 in do_exec
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 1072 in executetask
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 213 in run
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 277 in _perform_spawn
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 259 in integrate_as_primary_thread
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 1047 in serve
File "/home/jcazacu/Repos/main_repo/ares-game-server/.tox/test34/lib/python3.4/site-packages/execnet/gateway_base.py", line 1534 in serve
File "", line 8 in
File "", line 1 in
......[gw1] node down: Not properly terminated
fReplacing crashed slave gw1

Any ideas?

Currently we just moved the task tests to run serially since the failures are so unpredictable.

Start by reproducing the failure with pytest -n against tests using Celery tasks and @override_settings(task_always_eager=True), comparing parallel and serial runs with the reported Python, pytest-xdist, and execnet versions. Use the faulthandler traceback through execnet/gateway_base.py and Django's sqlite3 backend to narrow the failing boundary; done means parallel execution no longer produces a segmentation fault or “node down” termination.