pytest-dev / pytest-dev/pytest-xdist
Performance Degradation in Kubernetes with pytest-xdist
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
I’m experiencing significant performance degradation when running tests with pytest-xdist in a Kubernetes environment compared to running them locally.
Environment Details
Local Machine:
• CPU: 8 cores
• Memory: 16 GB
• Command: pytest -n 8 (all distributions are working well)
• Running Time: 10-15 minutes
• Screenshot:
Dedicated Machine on Kubernetes:
• Pod Configuration:
• CPU: 24 cores
• Memory: 48 GB
• Resources Requests and Limits: Correctly set
• Command: pytest -n 21 --dist=loadfile
• If using --dist=load or --dist=worksteal, workers are getting crushed and tests fail.
• Running Time: 35-57 minutes
• Screenshot:
Despite having significantly more resources in Kubernetes, the test running time is much longer, and using --dist=load or --dist=worksteal causes failures.
I need help understanding why this performance degradation is happening and how to optimize pytest-xdist in Kubernetes.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or failing test is named. Start by reproducing the local and Kubernetes commands, comparing --dist=loadfile with --dist=load and --dist=worksteal while checking the reported resource configuration. Done means identifying the cause of the slowdown or worker failures and documenting a validated optimization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- devops, performance, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100