KeyError in dsession.worker_internal_error() when a hook invokes config.notify_exception()

Using pytest-xdist 3.5.0 with pytest 7.4.3 (both are latest releases from PyPi at the time of this writing). Python 3.10.13 on OS X

KeyError in dsession.worker_internal_error() when a hook invokes config.notify_exception(): KeyError: <WorkerController gw0>

This appears to be a race condition.

To reproduce:

1. Install pytest and pytest-xdist
2. Create the file internalerror_plugin.py somewhere in PYTHONPATH with the following contents:

import pytest

@pytest.hookimpl(tryfirst=True)
def pytest_collect_file(file_path, parent):
  config=parent.config

  try:
    assert False
  except AssertionError:
    excinfo = pytest.ExceptionInfo.from_current()
    config.notify_exception(excinfo=excinfo)

3. Run: pytest -n 3 -p "internalerror_plugin" , which results in the following exception:

rootdir: /Users/vkruglik/xdist-debug
plugins: xdist-3.5.0
ready: 1/3 workers      INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/Users/vkruglik/xdist-debug/internalerror_plugin.py", line 20, in pytest_collect_file
INTERNALERROR> E                 assert False
INTERNALERROR> E             AssertionError: assert False
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> ../opensource/pytest-venv/lib/python3.10/site-packages/xdist/dsession.py:200: AssertionError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/xdist/dsession.py", line 123, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/xdist/dsession.py", line 148, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/xdist/dsession.py", line 198, in worker_internal_error
INTERNALERROR>     self._active_nodes.remove(node)
INTERNALERROR> KeyError: <WorkerController gw0>

4. With -n 1, I see a different exception:

$ pytest -n 1 -p "internalerror_plugin" 
======================================================================================================= test session starts ========================================================================================================
platform darwin -- Python 3.10.13, pytest-7.4.3, pluggy-1.3.0
rootdir: /Users/vkruglik/xdist-debug
plugins: xdist-3.5.0
ready: 1/1 worker      INTERNALERROR> def worker_internal_error(self, node, formatted_error):
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>     
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/Users/vkruglik/xdist-debug/internalerror_plugin.py", line 20, in pytest_collect_file
INTERNALERROR> E                 assert False
INTERNALERROR> E             AssertionError: assert False
INTERNALERROR> E           assert False
INTERNALERROR> 
INTERNALERROR> ../opensource/pytest-venv/lib/python3.10/site-packages/xdist/dsession.py:200: AssertionError
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/xdist/dsession.py", line 123, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/Users/vkruglik/opensource/pytest-venv/lib/python3.10/site-packages/xdist/dsession.py", line 137, in loop_once
INTERNALERROR>     raise RuntimeError("Unexpectedly no active workers available")
INTERNALERROR> RuntimeError: Unexpectedly no active workers available

Start with xdist/dsession.py at worker_internal_error and reproduce the failure using the internalerror_plugin.py example with pytest -n 3 and pytest -n 1. Trace how config.notify_exception() affects worker tracking and determine the expected handling when a worker is reported more than once. Done means the reproduction no longer raises KeyError or the unexpected-no-active-workers RuntimeError.