allure-framework / allure-framework/allure-python

Missing tests in report after pytest-timeout crashes the python process

Open
#641 1 comment 0 reactions 0 assignees View on GitHub
theme:pytest type:enhancement
Dominant language
Python
Stars
814
Forks
260
PR merge metrics
No merged PRs in 30d

Description

[//]: # (
. Note: for support questions, please use Stackoverflow or Gitter**.
. This repository's issues are reserved for feature requests and bug reports.
.
. In case of any problems with Allure Jenkins plugin** please use the following repository
. to create an issue: https://github.com/jenkinsci/allure-plugin/issues
.
. Make sure you have a clear name for your issue. The name should start with a capital
. letter and no dot is required in the end of the sentence. An example of good issue names:
.
. - The report is broken in IE11
. - Add an ability to disable default plugins
. - Support emoji in test descriptions
)

#### I'm submitting a ...
- [x] bug report
- [ ] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.

#### What is the current behavior?
When the timeout occurred (from pytest-timeout), the python process is stopping brutally (crashes), the current test and the tests that follow after don't appear in the report at all.

#### If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem
Running 4 tests with timeout of 8 seconds using pytest-timeout, while the 3rd test takes 10 seconds.
The execution stops brutally during the 3rd test and the allure report shows only 2 tests that passed.

This is the code of the tests:
import time

def test_this():
try:
time.sleep(0.1)
except Exception:
pass

def test_this3():
try:
time.sleep(0.1)
except Exception:
pass

def test_that():
time.sleep(10)
pass

def test_this2():
try:
time.sleep(0.1)
except Exception:
pass

#### What is the expected behavior?
The 3rd and the 4th tests should be in the report at least with (not tested or skipped status)

#### What is the motivation / use case for changing the behavior?
To avoid misleading that the execution passed successfully and also to keep the running suite with the same number of tests.

#### Please tell us about your environment:

- Allure version: 2.29.0
- Test framework: pytest@6.2.5
- Allure adaptor: allure-pytest@2.9.45

#### Other information

[//]: # (
. e.g. detailed explanation, stacktraces, related issues, suggestions
. how to fix, links for us to have more context, eg. Stackoverflow, Gitter etc
)

I think you should populate the report the remaining tests and the one that cause the timeout.

* I saw a PR in allure/android that fixed similar issue.

Stacktrace:
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of MainThread (59068) ~~~~~~~~~~~~~~~~~~~~~~~~~~
File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.3.3\plugins\python-ce\helpers\pycharm\_jb_pytest_runner.py", line 51, in
sys.exit(pytest.main(args, plugins_to_load + [Plugin]))
File "C:\Python37\lib\site-packages\_pytest\config\__init__.py", line 163, in main
config=config
File "C:\Python37\lib\site-packages\pluggy\hooks.py", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 87, in
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "C:\Python37\lib\site-packages\pluggy\callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "C:\Python37\lib\site-packages\_pytest\main.py", line 316, in pytest_cmdline_main
return wrap_session(config, _main)
File "C:\Python37\lib\site-packages\_pytest\main.py", line 269, in wrap_session
session.exitstatus = doit(config, session) or 0
File "C:\Python37\lib\site-packages\_pytest\main.py", line 323, in _main
config.hook.pytest_runtestloop(session=session)
File "C:\Python37\lib\site-packages\pluggy\hooks.py", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 87, in
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "C:\Python37\lib\site-packages\pluggy\callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "C:\Python37\lib\site-packages\_pytest\main.py", line 348, in pytest_runtestloop
item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
File "C:\Python37\lib\site-packages\pluggy\hooks.py", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 87, in
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "C:\Python37\lib\site-packages\pluggy\callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 109, in pytest_runtest_protocol
runtestprotocol(item, nextitem=nextitem)
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 126, in runtestprotocol
reports.append(call_and_report(item, "call", log))
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 215, in call_and_report
call = call_runtest_hook(item, when, **kwds)
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 255, in call_runtest_hook
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 311, in from_call
result: Optional[TResult] = func()
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 255, in
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File "C:\Python37\lib\site-packages\pluggy\hooks.py", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 87, in
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "C:\Python37\lib\site-packages\pluggy\callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "C:\Python37\lib\site-packages\_pytest\runner.py", line 162, in pytest_runtest_call
item.runtest()
File "C:\Python37\lib\site-packages\_pytest\python.py", line 1641, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "C:\Python37\lib\site-packages\pluggy\hooks.py", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "C:\Python37\lib\site-packages\pluggy\manager.py", line 87, in
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "C:\Python37\lib\site-packages\pluggy\callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "C:\Python37\lib\site-packages\_pytest\python.py", line 183, in pytest_pyfunc_call
result = testfunction(**testargs)
File "C:\GitRep\ng_fw_automation\fw_automation\test_try.py", line 24, in test_that
time.sleep(10)

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

Process finished with exit code 1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.