pytest-dev / pytest-dev/pytest-xdist

Performance Degradation in Kubernetes with pytest-xdist

Open
#1,108 0 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

I’m experiencing significant performance degradation when running tests with pytest-xdist in a Kubernetes environment compared to running them locally.

Environment Details

Local Machine:

•	CPU: 8 cores
•	Memory: 16 GB
•	Command: pytest -n 8 (all distributions are working well)
•	Running Time: 10-15 minutes
•	Screenshot: 

image

Dedicated Machine on Kubernetes:

•	Pod Configuration:
•	CPU: 24 cores
•	Memory: 48 GB
•	Resources Requests and Limits: Correctly set
•	Command: pytest -n 21 --dist=loadfile
•	If using --dist=load or --dist=worksteal, workers are getting crushed and tests fail.
•	Running Time: 35-57 minutes
•	Screenshot: 

Screenshot 2024-07-11 at 18 08 05

Despite having significantly more resources in Kubernetes, the test running time is much longer, and using --dist=load or --dist=worksteal causes failures.

I need help understanding why this performance degradation is happening and how to optimize pytest-xdist in Kubernetes.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or failing test is named. Start by reproducing the local and Kubernetes commands, comparing --dist=loadfile with --dist=load and --dist=worksteal while checking the reported resource configuration. Done means identifying the cause of the slowdown or worker failures and documenting a validated optimization.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops, performance, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.