test_workqueue_ordered_by_size is flaky

After upgrading pytest-xdist from 3.3.1 to 3.5.0 we are seeing the following test failure come up once in a while.

============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /build/pytest-xdist-3.5.0
configfile: tox.ini
testpaths: testing
plugins: xdist-3.5.0
24 workers [195 items]   m
s.x......................x.x...........................s................ [ 36%]
.........x................s..................F......x........x.xx....... [ 73%]
.....x.......s...........s..s............x..x......                      [100%]
=================================== FAILURES ===================================
_________________ TestLoadScope.test_workqueue_ordered_by_size _________________
[gw14] linux -- Python 3.11.6 /nix/store/myhqvbw8sknili2cn9av3pncg63rkd0p-python3-3.11.6/bin/python3.11

self = <acceptance_test.TestLoadScope object at 0x7ffff58baf10>
pytester = <Pytester PosixPath('/build/pytest-of-nixbld/pytest-0/popen-gw14/test_workqueue_ordered_by_size0')>

    def test_workqueue_ordered_by_size(self, pytester: pytest.Pytester) -> None:
        test_file = """
            import pytest
            @pytest.mark.parametrize('i', range({}))
            def test(i):
                pass
        """
        pytester.makepyfile(test_a=test_file.format(10), test_b=test_file.format(20))
        result = pytester.runpytest("-n2", "--dist=loadscope", "-v")
>       assert get_workers_and_test_count_by_prefix(
            "test_a.py::test", result.outlines
        ) == {"gw1": 10}
E       AssertionError: assert {'gw0': 10} == {'gw1': 10}
E         Left contains 1 more item:
E         {'gw0': 10}
E         Right contains 1 more item:
E         {'gw1': 10}
E         Use -v to get more diff

/build/pytest-xdist-3.5.0/testing/acceptance_test.py:1244: AssertionError

Open testing/acceptance_test.py and inspect TestLoadScope.test_workqueue_ordered_by_size together with get_workers_and_test_count_by_prefix. Reproduce the failure with pytest -n2 --dist=loadscope -v; done means the test passes reliably without the reported worker-assignment flake.