pytest-dev / pytest-dev/pytest-timeout

Timeout with xdist causes worker crash and reported as error in html

Open
#137 5 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
258
Forks
69
No merged PRs in 30d

Description

Hello! Is this worker crash fully fixed in 2.1.0?
I am still having this problem using pytest-xdist==2.5.0 and pytest-timeout==2.1.0
env

plugins: html-3.1.1, parallel-0.1.1, metadata-1.11.0, timeout-2.1.0, forked-1.4.0, xdist-2.5.0, assume-2.4.3
timeout: 60.0s
timeout method: thread
timeout func_only: False

stacks

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

~~~~~~~~~~~~~~~~~~~~~ Stack of <unknown> (139896081934080) ~~~~~~~~~~~~~~~~~~~~~
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 285, in _perform_spawn
    reply.run()
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 220, in run
    self._result = func(*args, **kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 967, in _thread_receiver
    msg = Message.from_io(io)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 432, in from_io
    header = io.read(9)  # type 1, channel 4, payload 4
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 400, in read
    data = self._read(numbytes - len(buf))

~~~~~~~~~~~~~~~~~~~~ Stack of MainThread (139896100013888) ~~~~~~~~~~~~~~~~~~~~~
  File "<string>", line 1, in <module>
  File "<string>", line 8, in <module>
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 1554, in serve
    WorkerGateway(io=io, id=id, _startcount=2).serve()
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 1060, in serve
    self._execpool.integrate_as_primary_thread()
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 267, in integrate_as_primary_thread
    self._perform_spawn(reply)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 285, in _perform_spawn
    reply.run()
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 220, in run
    self._result = func(*args, **kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/execnet/gateway_base.py", line 1084, in executetask
    do_exec(co, loc)  # noqa
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/xdist/remote.py", line 291, in <module>
    config.hook.pytest_cmdline_main(config=config)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/main.py", line 316, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/main.py", line 269, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/main.py", line 323, in _main
    config.hook.pytest_runtestloop(session=session)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/xdist/remote.py", line 91, in pytest_runtestloop
    self.run_one_test(torun)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/xdist/remote.py", line 110, in run_one_test
    self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 109, in pytest_runtest_protocol
    runtestprotocol(item, nextitem=nextitem)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 126, in runtestprotocol
    reports.append(call_and_report(item, "call", log))
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 215, in call_and_report
    call = call_runtest_hook(item, when, **kwds)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 255, in call_runtest_hook
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 311, in from_call
    result: Optional[TResult] = func()
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 255, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/runner.py", line 162, in pytest_runtest_call
    item.runtest()
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/python.py", line 1641, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/home/name/test/var/.tox/py37/lib/python3.7/site-packages/_pytest/python.py", line 183, in pytest_pyfunc_call
    result = testfunction(**testargs)
  File "/home/name/test/e2e/cases/outer/test_machine.py", line 43, in test_zero
    time.sleep(60)

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

[gw11] node down: Not properly terminated
[gw11] [ 99%] FAILED e2e/cases/outer/test_machine.py::Testfile::test_zero[xxxxxxxxx] 

replacing crashed worker gw11
[gw12] linux Python 3.7.3 cwd: /home/name/test                                                           
[gw12] Python 3.7.3 (default, Jan 22 2021, 20:04:44)  -- [GCC 8.3.0]                                              
gw0 [858] / gw1 [858] / gw2 [858] / gw3 [858] / gw4 [858] / gw5 [858] / gw6 [858] / gw12 [858] / gw8 [858] / gw9 [858]
e2e/cases/outer/test_machine.py::Testfile::test_1kb[xxxxxxxx] 
[gw12] [100%] PASSED e2e/cases/outer/test_machine.py::Testfile::test_1kb[xxxxxxx]

Originally posted by @AZ-201 in https://github.com/pytest-dev/pytest-timeout/issues/88#issuecomment-1188540109

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the timeout in e2e/cases/outer/test_machine.py with pytest-timeout 2.1.0 and pytest-xdist 2.5.0, using the reported plugin configuration. Read the worker-crash output around the timeout and determine whether the behavior remains; done means the reproduction no longer reports an improperly terminated worker as an HTML error, with a regression test covering the result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.