[DX] choose optimal job size to reach best speed

13 111 files

279 files (a single module of the same project)

On 279 files even a single worker - jobSize: 300, no parallelism at all - beats the current default running on 14, because starting 14 processes costs more than the parallelism returns.

The optimum does move with corpus size (~50 for 279 files, 300+ for 13 111), but 100 is close to both, and anything in 100-200 beats 16 everywhere I measured.

So yes, increasing default job size to 100 might be a good idea, but:

jobSize and timeoutSeconds interact. A worker has to finish a whole chunk inside the per-job timeout, and 16 -> 150 multiplies the work per chunk by ~10. With a heavy rule set and a cold cache I hit exactly that - Child process timed out after 120 seconds, exit 1 - at jobSize: 150, where 16 completed. If the default goes up, the default timeout likely needs to go up with it, otherwise the failure mode moves from "slow" to "red build".

Originally posted by @webard in https://github.com/rectorphp/rector-src/issues/8489#issuecomment-5662569618

Start by tracing where the default jobSize and timeoutSeconds values are defined and how worker chunks are scheduled. Reproduce the reported benchmarks for small and large file sets, including the timeout case with heavier rules. Done means choosing and documenting defaults that improve throughput without causing the reported child-process timeouts.