ethereum / ethereum/execution-specs
Dynamically assign number of pytest-xdist workers via `pytest_xdist_auto_num_workers`
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 505
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 116
Description
It should be possible with [`pytest_xdist_auto_num_workers`](https://github.com/pytest-dev/pytest-xdist/blob/8b60b1ef5d48974a1cb69bc1a9843564bdc06498/src/xdist/plugin.py#L16) to:
- Read numbers of cores
- Read total system memory
- Determine if CPython or PyPy
And come up with a reasonable number of cores.
_Originally posted by @SamWilsn in https://github.com/ethereum/execution-specs/issues/2120#issuecomment-3843476312_
*[pytest docs: CPU parallelization](https://pytest-xdist.readthedocs.io/en/stable/distribution.html)*
Right now pytest uses the `-n auto` flag that directs pytest-xdist to use as many processes as the computer running the tests has **physical** CPU cores, up to the limit set by `--maxprocesses`. The maximum number of processes is currently set to specific values based on our experience of the CI machine's performance.
We could likely increase throughput by switching to the `-n logical` flag which would use the max number of **logical** CPU cores, however, before we do that we need to dynamically adjust the `--maxprocesses` cap based on the system that is running the tests. To do that, we will have to figure out how to determine the available memory on the system and properly adjust to ensure we don't run out.
We should also investigate using the `--dist worksteal` flag for impact on test times which could handle re-assigning jobs to workers who have completed their test queue more efficiently.
Contributor guide
Assessment
This issue has not been assessed yet.