pytest-dev / pytest-dev/pytest-xdist

Replacing crashed worker ends with failure and missing coverage

Open
#466 12 comments 32 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

First at all, thanks for your time and the develop of this project.

I've an issue where sometimes our CI system kills a thread (not python issue) and the affected
worker is restarted

May I'm interpreting in a bad way the auto worker restart feature.

What I expect is if the worker can be recovered then the test finalize without errors but in this case what I see a test error is raised and also coverage plugin is not able to recover coverage from the restarted worker

============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-5.1.1, py-1.8.0, pluggy-0.12.0
Django settings: settings.test (from command line option)
rootdir: /root/app/src, inifile: setup.cfg, testpaths: apps
plugins: responses-0.4.0, xdist-1.29.0, celery-4.3.0, profiling-1.7.0, cov-2.7.1, forked-1.0.2, django-3.5.1
gw0 [3154] / gw1 [3154] / gw2 [3154] / gw3 [3154] / gw4 [3154] / gw5 [3154] / gw6 [3154] / gw7 [3154]
[gw7] node down: Not properly terminated
f
replacing crashed worker gw7
gw0 [3154] / gw1 [3154] / gw2 [3154] / gw3 [3154] / gw4 [3154] / gw5 [3154] / gw6 [3154] / gw8 ok. [ 0%]
...............................
gw0 [3154] / gw1 [3154] / gw2 [3154] / gw3 [3154] / gw4 [3154] / gw5 [3154] / gw6 [3154] / gw8 [3154]
.............
=================================== FAILURES ===================================
____________ test/test_models.py ____________
[gw7] linux -- Python 3.7.4 /usr/local/bin/python
worker 'gw7' crashed while running 'tests/test_models.py::Test::test_xxx'
---- generated xml file: /root/app/src/test_result.xml -----

---------------------- coverage: failed slaves -----------------------
The following slaves failed to return coverage data, ensure that pytest-cov is installed on these laves.
w7

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the worker failure around tests/test_models.py::Test::test_xxx using the reported pytest, pytest-xdist, and pytest-cov setup. Then trace xdist's crashed-worker replacement behavior and pytest-cov's failed-slave handling. Done means a recovered worker does not fail the test session unexpectedly and coverage data is retained or the expected limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.