pytest-dev / pytest-dev/pytest-html

Pytest always outputs traceback for XFAIL [NOTRUN] test

Open
#118 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement help wanted
Dominant language
Python
Stars
779
Forks
260
PR merge metrics
No merged PRs in 30d

Description

Hi!
Is there anybody who can elaborate what is going wrong with pytest-html, which generates results for XFailed tests on 'setup' without running the actual test? It is always printing this traceback in details:

self = <CallInfo when='setup' exception: [NOTRUN] Failed due to changes in functionality>, func = <function call_runtest_hook.<locals>.<lambda> at 0x7fc6c50d9f28>, when = 'setup'

    def __init__(self, func, when):
        #: context of invocation: one of "setup", "call",
        #: "teardown", "memocollect"
        self.when = when
        self.start = time()
        try:
>           self.result = func()

.venv/lib/python3.5/site-packages/_pytest/runner.py:163: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.venv/lib/python3.5/site-packages/_pytest/runner.py:151: in <lambda>
    return CallInfo(lambda: ihook(item=item, **kwds), when=when)
.venv/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:745: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
.venv/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:339: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
.venv/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:334: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
.venv/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:613: in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
.venv/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:254: in _wrapped_call
    return call_outcome.get_result()
.venv/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:279: in get_result
    raise ex[1].with_traceback(ex[2])
.venv/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:265: in __init__
    self.result = func()
.venv/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:614: in execute
    res = hook_impl.function(*args)
.venv/lib/python3.5/site-packages/_pytest/skipping.py:190: in pytest_runtest_setup
    check_xfail_no_run(item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

item = <Function 'test_send_sync_with_distinct_sessions'>

    def check_xfail_no_run(item):
        """check xfail(run=False)"""
        if not item.config.option.runxfail:
            evalxfail = item._evalxfail
            if evalxfail.istrue():
                if not evalxfail.get('run', True):
>                   pytest.xfail("[NOTRUN] " + evalxfail.getexplanation())
E                   _pytest.skipping.XFailed: [NOTRUN] Failed due to changes in functionality

.venv/lib/python3.5/site-packages/_pytest/skipping.py:208: XFailed

Mainly, I just wanna to see only the reason, which is XFailed: [NOTRUN] Failed due to changes in functionality, same as it works for SKIPPED tests

Thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce an XFAIL(run=False) test in a pytest HTML report and compare its output with a skipped test. Start with the XFailed handling shown in _pytest/skipping.py and trace how the plugin renders setup outcomes. Done means the report shows only the XFailed reason instead of the full traceback.

Written by the indexing model from the issue text.

Assessment

Tech stack
html, python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.