loadscope/loadgroup: a crashed worker's COMPLETED scopes are re-queued as empty work units and can deadlock the session at the end of the run

pytest-xdist 3.8.0, pytest 9.0.3, execnet 2.1.2, Windows 11, -n 4 --dist loadgroup, ~12,700 tests.

When a worker dies mid-run (in our case pytest-timeout's thread method killing the worker after a slow test), LoadScopeScheduling.remove_node puts the node's ENTIRE workload back on the queue:

# Made uncompleted work unit available again
self.workqueue.update(workload)

workload still contains every scope the dead node had already completed (all values True). Those scopes are later popped by _assign_work_unit, which computes nodeids_indexes from the not-completed items — an empty list — and calls node.send_runtest_some([]). The worker enqueues nothing, so it never produces a runtest_protocol_complete for that unit, and since a node is only rescheduled from mark_test_complete, the empty unit has consumed one reschedule opportunity for nothing.

A few empty units are absorbed (the node's later completions reschedule past them). But the dead node's completed scopes are appended at the END of the ordered queue, so the tail of the run is nothing but empty units; when there are more of them than the survivors have completions left, every worker parks in run_one_test waiting for a next item or shutdown, tests_finished is False (workqueue non-empty), no further events arrive, and DSession.loop_once waits forever. The replacement worker spawned by worker_errordown is fed the same empty unit first, so it never runs anything either.

Observed with a controller-side trace: at the crash the queue jumped from 1 to 6143 units (the dead worker's half-run of finished scopes); the final state was workqueue=6, one pending (unrun, held) test per node, six runtests commands with indices=[], and all workers idle in TestQueue.get. The visible symptom is a run that reaches ~99% and never prints a summary. Seen again twice on 2026-09-01 on the same setup.

Minimal reproduction with the scheduler class and fake nodes (2 nodes, 20 two-test scopes; the "dead" node completes 14 tests then is removed; a replacement node is added and schedule() re-run; survivors complete everything they are sent): upstream ends with tests_finished == False, three units still queued, six empty runtests sends. Filtering workload to units with pending items in remove_node (and refusing to send an empty unit in _assign_work_unit) makes the same scenario finish with every remaining test run exactly once.

Suggested fix:

self.workqueue.update(
    (scope, unit) for scope, unit in workload.items()
    if any(not done for done in unit.values()))

plus a guard in _assign_work_unit to skip units with nothing pending. Happy to open a PR with the scenario as a test.

Adjacent reports that look like different mechanisms: #466 (replacement worker ends with a failure and missing coverage) and #932 (tests reported both FAILED and PASSED after a node crash); this one is specifically the completed-scope re-queue under loadscope/loadgroup and the end-of-run hang it produces.

Start with LoadScopeScheduling.remove_node and _assign_work_unit, then reproduce the described scheduler scenario with fake nodes: a dead node completes tests before removal and a replacement is added. Add coverage for loadscope/loadgroup scheduling and verify that completed scopes are not re-queued, empty work units are not sent, and the run finishes with every remaining test executed exactly once.