pytest-dev / pytest-dev/pytest-xdist
Tests show as both FAILED and PASSED after node crash
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
Hi,
I'm noticing that when we have a worker crash (which started happening frequenctly last week) the test appears as both FAILED and PASSED.
E.x.: Below is a snippet of the logs for tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py::StoredProcTestCase::test_updates_before_start_date_ignored@/opt/gitlab-runner/builds/abc/pipelines/tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py:21. It ran on worker #7 which crashed and then ran later on worker #8 which PASSED.
Line 4026: tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py::StoredProcTestCase::test_updates_before_start_date_ignored@/opt/gitlab-runner/builds/abc/pipelines/tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py:21
Line 4037: [gw7] node down: Not properly terminated
Line 4038: [gw7] [ 97%] FAILED tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py::StoredProcTestCase::test_updates_before_start_date_ignored@/opt/gitlab-runner/builds/abc/pipelines/tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py:21
replacing crashed worker gw7
Line 4047: tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py::StoredProcTestCase::test_updates_before_start_date_ignored@/opt/gitlab-runner/builds/abc/pipelines/tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py:21
Line 4070: [gw8] [ 98%] PASSED tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py::StoredProcTestCase::test_updates_before_start_date_ignored@/opt/gitlab-runner/builds/abc/pipelines/tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py:21
Line 7994: worker 'gw7' crashed while running 'tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py::StoredProcTestCase::test_updates_before_start_date_ignored@/opt/gitlab-runner/builds/abc/pipelines/tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py:21'
Line 8220: =========================== short test summary info ============================
Line 8227: FAILED tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py::StoredProcTestCase::test_updates_before_start_date_ignored@/opt/gitlab-runner/builds/abc/pipelines/tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py:21
Line 9397: PASSED tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py::StoredProcTestCase::test_updates_before_start_date_ignored@/opt/gitlab-runner/builds/abc/pipelines/tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py:21
Line 9420: = 6 failed, 1137 passed, 252 skipped, 96 warnings, 15 rerun in 6617.65s (1:50:17) =
package versions:
pytest-7.4.0
pytest_cov-4.1.0
pytest_xdist-3.3.1
coverage-7.2.7
pytest_rerunfailures-12.0
psutil-5.9.5
command line:
pytest --log-format="%Y-%m-%dT%H:%M:%S.%f%z" --log-date-format="%Y-%m-%d %H:%M:%S" --log-format "%(asctime)s %(levelname)-8s [%(name)s|%(process)d|%(thread)d|%(threadName)s] [%(pathname)s:%(funcName)s:%(lineno)d] %(message)s" --max-worker-restart 5 -n 16 --dist loadgroup -rfEsxXp --reruns 2 --reruns-delay 30 -v --tb=long -o faulthandler_timeout=3600 --durations=20 --durations-min=60 --cov=src/ tests
We're running about 1000 tests using 16 workers
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported pytest command and the worker-crash output for tests/cimdb/staging/sp/xyz/test_portfolio_valuation_lot_merge.py::StoredProcTestCase::test_updates_before_start_date_ignored. Reproduce the crash with pytest-xdist using multiple workers, then inspect how the crashed worker and replacement worker are reflected in the final summary. Done means a single test does not appear as both FAILED and PASSED after a worker crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100