pytest-dev / pytest-dev/pytest-xdist
Replacing crashed worker ends with failure and missing coverage
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
First at all, thanks for your time and the develop of this project.
I've an issue where sometimes our CI system kills a thread (not python issue) and the affected
worker is restarted
May I'm interpreting in a bad way the auto worker restart feature.
What I expect is if the worker can be recovered then the test finalize without errors but in this case what I see a test error is raised and also coverage plugin is not able to recover coverage from the restarted worker
============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.1, py-1.8.0, pluggy-0.12.0
Django settings: settings.test (from command line option)
rootdir: /root/app/src, inifile: setup.cfg, testpaths: apps
plugins: responses-0.4.0, xdist-1.29.0, celery-4.3.0, profiling-1.7.0, cov-2.7.1, forked-1.0.2, django-3.5.1
gw0 [3154] / gw1 [3154] / gw2 [3154] / gw3 [3154] / gw4 [3154] / gw5 [3154] / gw6 [3154] / gw7 [3154]
[gw7] node down: Not properly terminated
f
replacing crashed worker gw7
gw0 [3154] / gw1 [3154] / gw2 [3154] / gw3 [3154] / gw4 [3154] / gw5 [3154] / gw6 [3154] / gw8 ok. [ 0%]
...............................
gw0 [3154] / gw1 [3154] / gw2 [3154] / gw3 [3154] / gw4 [3154] / gw5 [3154] / gw6 [3154] / gw8 [3154]
.............
=================================== FAILURES ===================================
____________ test/test_models.py ____________
[gw7] linux -- Python 3.7.4 /usr/local/bin/python
worker 'gw7' crashed while running 'tests/test_models.py::Test::test_xxx'
---- generated xml file: /root/app/src/test_result.xml -----
---------------------- coverage: failed slaves -----------------------
The following slaves failed to return coverage data, ensure that pytest-cov is installed on these laves.
w7
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the worker failure around tests/test_models.py::Test::test_xxx using the reported pytest, pytest-xdist, and pytest-cov setup. Then trace xdist's crashed-worker replacement behavior and pytest-cov's failed-slave handling. Done means a recovered worker does not fail the test session unexpectedly and coverage data is retained or the expected limitation is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100