pytest-dev / pytest-dev/pytest

pytest leaks memory and crashes jenkins slaves when this error occurs

Open
#6,670 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

plugin: xdist type: performance
Dominant language
Python
Stars
14.5k
Forks
3.4k
Avg merge
2d 9h
Merged PRs (30d)
35

Description

Hi team, unfortunately I don't have exact steps to recreate this, but I will include as much information as I have at this point and build upon it.

pytest version: 5.2.1

[gw11] linux Python 3.7.5 cwd: /home/jenkins/workspace/branch/rootdir
18:16:25  /home/jenkins/workspace/branch/rootdir/venv/lib/python3.7/site-packages/_pytest/pathlib.py:47: PytestWarning: (rm_rf) error removing /home/jenkins/workspace/branch/rootdir/.pytest_cache/v: [Errno 2] No such file or directory: 'v'
18:16:25    PytestWarning("(rm_rf) error removing {}: {}".format(path, excvalue))
18:16:25  /home/jenkins/workspace/branch/rootdir/venv/lib/python3.7/site-packages/_pytest/pathlib.py:47: PytestWarning: (rm_rf) error removing /home/jenkins/workspace/branch/rootdir/.pytest_cache/.gitignore: [Errno 2] No such file or directory: '.gitignore'
18:16:25    PytestWarning("(rm_rf) error removing {}: {}".format(path, excvalue))
18:16:34  collected 1947 items / 987 deselected / 960 selected

Not sure if this is caused by xdist? not sure if its also one of our plugins (internally loaded) but it looks like core pytest code here. Weird that it is looking for .gitignore etc to try and rm?

I will update this ticket as more information comes to light, sorry for lack of information, it happens perhaps 1 in 30 runs on jenkins.

When the above occurs, memory on the jenkins slave is consumed and consumed eventually reaching 100% and destroying the node, leaving the build hanging indefinitely

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the warning at _pytest/pathlib.py:47 in pytest 5.2.1 and compare runs involving xdist, the internal plugins, and Jenkins. Reproduce the intermittent failure if possible, then trace the memory growth associated with the repeated rm_rf warnings; done means identifying a reproducible cause and preventing the Jenkins slave from exhausting memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.