total worker initialization time scales linearly to the number of workers

It seems that currently, initializing workers is blocking and done sequentially:

    def setup_nodes(self, putevent):
        self.config.hook.pytest_xdist_setupnodes(config=self.config, specs=self.specs)
        self.trace("setting up nodes")
        # from multiprocessing import Pool
        # p = Pool(len(self.specs))
        # nodes = p.map(lambda spec: self.setup_node(spec, putevent), self.specs)
        nodes = []
        for spec in self.specs:
            nodes.append(self.setup_node(spec, putevent))
        return nodes

This means that starting a large number of workers even on a single machine with a large number of cpu cores takes a long time. For example, it takes about 45 seconds to start one worker per cpu core on my machine with 88 cores. The effect is even more extreme when workers are being started on remote machines with additional network latency added.

With the advent of a larger number of cores per machine and more frequent access to large clusters of machines, it would be nice to quickly horizontally scale to a large number of workers, even in cases where this would be the difference between 5 minutes on 8 workers versus 8 seconds on 300 workers. Obviously it isn't quite that simple, but there is clearly room theoretically for improvement.

As you can see from the code posted above, i've tried using multiprocessing to paralyze the setup of new nodes, however, the execnet objects used aren't pickleable so this naïve solution does not work.

I've also spent a little bit of time investigating the use of something like ray for rapid distribution across an admittedly homogenous cluster, but i ran into trouble where Function objects were not pickleable even by dill and cloudpickle.

Has anyone looked into how else this problem could be solved? Perhaps my identification of the problem is also incorrect. Are there other critical factors that are preventing the use of a large number of workers?

Start at the setup_nodes entry point shown in the issue and trace setup_node and the execnet objects involved in worker initialization. Determine where sequential startup is imposed and define success as safe initialization that reduces scaling time for both local and remote workers without breaking existing setup behavior.