ethereum / ethereum/execution-specs

Dynamically assign number of pytest-xdist workers via `pytest_xdist_auto_num_workers`

Open
#1,393 5 comments 0 reactions 0 assignees View on GitHub
A-spec-tools C-feat E-easy P-low stale
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 14h
Merged PRs (30d)
116

Description

It should be possible with [`pytest_xdist_auto_num_workers`](https://github.com/pytest-dev/pytest-xdist/blob/8b60b1ef5d48974a1cb69bc1a9843564bdc06498/src/xdist/plugin.py#L16) to:

- Read numbers of cores
- Read total system memory
- Determine if CPython or PyPy

And come up with a reasonable number of cores.

_Originally posted by @SamWilsn in https://github.com/ethereum/execution-specs/issues/2120#issuecomment-3843476312_

*[pytest docs: CPU parallelization](https://pytest-xdist.readthedocs.io/en/stable/distribution.html)*

Right now pytest uses the `-n auto` flag that directs pytest-xdist to use as many processes as the computer running the tests has **physical** CPU cores, up to the limit set by `--maxprocesses`. The maximum number of processes is currently set to specific values based on our experience of the CI machine's performance.

We could likely increase throughput by switching to the `-n logical` flag which would use the max number of **logical** CPU cores, however, before we do that we need to dynamically adjust the `--maxprocesses` cap based on the system that is running the tests. To do that, we will have to figure out how to determine the available memory on the system and properly adjust to ensure we don't run out.

We should also investigate using the `--dist worksteal` flag for impact on test times which could handle re-assigning jobs to workers who have completed their test queue more efficiently.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.