Worker startup performances

I have a computer with a AMD Ryzen 7 1800X with 8 cores (16 logical CPUs) and 16Go of ram. My project is a flask website with ~400 tests, most of them testing the generated pages with webtest, or the models with an in-memory temporary database. There is almost no i/o.

Here are the results for launching one or all my tests, with -n0, -n1 or -nauto. The one test duration is ~0.7s, and this is the slowest. All the tests pass. In the following table there is the data displayed by pytest, then the data displayed by the time command (i.e. time pytest -nauto).

What I read in those data:

• There is a difference of ~4s between the duration announced by pytest, and the duration announced by the system. I suppose it is not really pytest-xdist, but what is the difference due to?
• Passing from -n0 to -n1 costs between ~3.5s and ~5s. I understand that spawning processes can be costly, but 3.5s seems a lot, especially when only a few tests are ran.
• With 1 test, -n16 is ~8s slower than -n1. As 15 workers won't run any test, I guess that the 8 additional seconds are lost by spawning the 15 useless workers. By why spawning n workers is so much longer than spawning a single worker? Maybe related to #272
• With 400 tests -n16 is not 16 times faster than -n1, but only ~2.7 times faster. I understand that there will never be a 16x improvement, but 2.7 is a bit disappointing. Maybe related to #657

I found the most annoying point is the spending of 3.5s for spawning the first worker. Is there some way this could be accelerated (either by configuration, or by a patch)?

For instance, what would you think of some kind of pytest agent? That would be: a pytest agent running in the background, that would run idling workers, and when a user would launch tests, they would be sent to the workers, and the workers would not stop after.

What do you think?

Start by reproducing the reported pytest -n0, -n1, and -nauto timing comparisons for one and roughly 400 tests. Investigate worker startup and scheduling overhead, then document a concrete optimization or agent design with benchmark results showing what is improved.