pytest-dev / pytest-dev/pytest-xdist

Internal error when slave crashed during test

Open
#204 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

This happens rarely when the slave receives a signal. Doesn't seem to have anything to do with a specific test, but just randomly. It's possible that this is an interaction with pytest-timeout, which causes a crash in pytest_runtest_protocol.

When this happens, the state of the session seems to become corrupted, as the master does not note a failed test and continues on, reporting no errors, and executing only a small subset of all tests.

A full log is available at: https://travis-ci.org/borgbackup/borg/jobs/260118022

platform linux -- Python 3.5.3, pytest-3.2.0, py-1.4.34, pluggy-0.4.0 -- /home/travis/build/borgbackup/borg/.tox/py35/bin/python3.5
plugins: xdist-1.18.2, timeout-1.2.0, faulthandler-1.3.1, cov-2.5.1, benchmark-3.1.1

.tox/py35/lib/python3.5/site-packages/borg/testsuite/archive.py::test_key_length_msgpacked_items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/main.py", line 110, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/main.py", line 146, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 254, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "<remote exec>", line 61, in pytest_runtestloop
INTERNALERROR>   File "<remote exec>", line 77, in run_tests
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 254, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 245, in _wrapped_call
INTERNALERROR>     next(wrap_controller)   # first yield
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/pytest_timeout.py", line 76, in pytest_runtest_protocol
INTERNALERROR>     timeout_setup(item)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/pytest_timeout.py", line 115, in timeout_setup
INTERNALERROR>     signal.signal(signal.SIGALRM, handler)
INTERNALERROR>   File "/opt/python/3.5.3/lib/python3.5/signal.py", line 47, in signal
INTERNALERROR>     handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
INTERNALERROR> ValueError: signal only works in main thread

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/main.py", line 110, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/main.py", line 146, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
INTERNALERROR>     wrap_controller.send(call_outcome)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/xdist/dsession.py", line 539, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/xdist/dsession.py", line 558, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/home/travis/build/borgbackup/borg/.tox/py35/lib/python3.5/site-packages/xdist/dsession.py", line 596, in slave_slavefinished
INTERNALERROR>     assert not crashitem, (crashitem, node)
INTERNALERROR> AssertionError: ('.tox/py35/lib/python3.5/site-packages/borg/testsuite/archive.py::ArchiveTimestampTestCase::test_with_microseconds', <SlaveController gw1>)
INTERNALERROR> assert not '.tox/py35/lib/python3.5/site-packages/borg/testsuite/archive.py::ArchiveTimestampTestCase::test_with_microseconds'

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the traceback in xdist/dsession.py around slave_slavefinished and loop_once, then examine how the pytest_timeout.py pytest_runtest_protocol failure affects the worker session. Reproduce the signal-related crash using the versions and plugins listed in the issue. Done means a crashed slave is reported as a failed test or session and the remaining suite is not silently skipped.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.