pytest-dev / pytest-dev/pytest
Test regressions in PyPy3.9 7.3.8
Open
Nobody has claimed this yet.
type: selftests
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
- a detailed description of the bug or problem you are having
- output of
pip listfrom the virtual environment you are using - pytest and operating system versions
- minimal example if possible (n/a)
Description
When running the test suite against PyPy 3.9 (v7.3.8), the following new test failures occur (via tox -e pypy3):
=================================== FAILURES ===================================
____________________ test_errors_in_xfail_skip_expressions _____________________
pytester = <Pytester PosixPath('/tmp/pytest-of-mgorny/pytest-1/test_errors_in_xfail_skip_expressions0')>
def test_errors_in_xfail_skip_expressions(pytester: Pytester) -> None:
pytester.makepyfile(
"""
import pytest
@pytest.mark.skipif("asd")
def test_nameerror():
pass
@pytest.mark.xfail("syntax error")
def test_syntax():
pass
def test_func():
pass
"""
)
result = pytester.runpytest()
markline = " ^"
pypy_version_info = getattr(sys, "pypy_version_info", None)
if pypy_version_info is not None and pypy_version_info < (6,):
markline = markline[5:]
elif sys.version_info >= (3, 8) or hasattr(sys, "pypy_version_info"):
markline = markline[4:]
if sys.version_info[:2] >= (3, 10):
expected = [
"*ERROR*test_nameerror*",
"*asd*",
"",
"During handling of the above exception, another exception occurred:",
]
else:
expected = [
"*ERROR*test_nameerror*",
]
expected += [
"*evaluating*skipif*condition*",
"*asd*",
"*ERROR*test_syntax*",
"*evaluating*xfail*condition*",
" syntax error",
markline,
"SyntaxError: invalid syntax",
"*1 pass*2 errors*",
]
> result.stdout.fnmatch_lines(expected)
E Failed: nomatch: '*ERROR*test_nameerror*'
E and: '============================= test session starts =============================='
E and: 'platform linux -- Python 3.9.10[pypy-7.3.8-final], pytest-7.2.0.dev12+g22756c28e, pluggy-1.0.0'
E and: 'rootdir: /tmp/pytest-of-mgorny/pytest-1/test_errors_in_xfail_skip_expressions0'
E and: 'collected 3 items'
E and: ''
E and: 'test_errors_in_xfail_skip_expressions.py EE. [100%]'
E and: ''
E and: '==================================== ERRORS ===================================='
E fnmatch: '*ERROR*test_nameerror*'
E with: '_______________________ ERROR at setup of test_nameerror _______________________'
E nomatch: '*evaluating*skipif*condition*'
E and: "name 'asd' is not defined"
E and: ''
E and: 'During handling of the above exception, another exception occurred:'
E fnmatch: '*evaluating*skipif*condition*'
E with: "Error evaluating 'skipif' condition"
E fnmatch: '*asd*'
E with: ' asd'
E nomatch: '*ERROR*test_syntax*'
E and: "NameError: name 'asd' is not defined"
E fnmatch: '*ERROR*test_syntax*'
E with: '________________________ ERROR at setup of test_syntax _________________________'
E nomatch: '*evaluating*xfail*condition*'
E and: 'invalid syntax. Perhaps you forgot a comma? (<xfail condition>, line 1)'
E and: ''
E and: 'During handling of the above exception, another exception occurred:'
E fnmatch: '*evaluating*xfail*condition*'
E with: "Error evaluating 'xfail' condition"
E exact match: ' syntax error'
E nomatch: ' ^'
E and: ' ^'
E and: 'SyntaxError: invalid syntax'
E and: '=========================== short test summary info ============================'
E and: 'ERROR test_errors_in_xfail_skip_expressions.py::test_nameerror'
E and: 'ERROR test_errors_in_xfail_skip_expressions.py::test_syntax'
E and: '========================= 1 passed, 2 errors in 0.02s =========================='
E remains unmatched: ' ^'
/tmp/pytest/testing/test_skipping.py:1171: Failed
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.9.10[pypy-7.3.8-final], pytest-7.2.0.dev12+g22756c28e, pluggy-1.0.0
rootdir: /tmp/pytest-of-mgorny/pytest-1/test_errors_in_xfail_skip_expressions0
collected 3 items
test_errors_in_xfail_skip_expressions.py EE. [100%]
==================================== ERRORS ====================================
_______________________ ERROR at setup of test_nameerror _______________________
name 'asd' is not defined
During handling of the above exception, another exception occurred:
Error evaluating 'skipif' condition
asd
NameError: name 'asd' is not defined
________________________ ERROR at setup of test_syntax _________________________
invalid syntax. Perhaps you forgot a comma? (<xfail condition>, line 1)
During handling of the above exception, another exception occurred:
Error evaluating 'xfail' condition
syntax error
^
SyntaxError: invalid syntax
=========================== short test summary info ============================
ERROR test_errors_in_xfail_skip_expressions.py::test_nameerror
ERROR test_errors_in_xfail_skip_expressions.py::test_syntax
========================= 1 passed, 2 errors in 0.02s ==========================
_______________________________ test_unraisable ________________________________
pytester = <Pytester PosixPath('/tmp/pytest-of-mgorny/pytest-1/test_unraisable0')>
@pytest.mark.filterwarnings("default::pytest.PytestUnraisableExceptionWarning")
def test_unraisable(pytester: Pytester) -> None:
pytester.makepyfile(
test_it="""
class BrokenDel:
def __del__(self):
raise ValueError("del is broken")
def test_it():
obj = BrokenDel()
del obj
def test_2(): pass
"""
)
result = pytester.runpytest()
assert result.ret == 0
> assert result.parseoutcomes() == {"passed": 2, "warnings": 1}
E AssertionError: assert {'passed': 2} == {'passed': 2, 'warnings': 1}
E Omitting 1 identical items, use -vv to show
E Right contains 1 more item:
E {'warnings': 1}
E Use -v to get more diff
/tmp/pytest/testing/test_unraisableexception.py:28: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.9.10[pypy-7.3.8-final], pytest-7.2.0.dev12+g22756c28e, pluggy-1.0.0
rootdir: /tmp/pytest-of-mgorny/pytest-1/test_unraisable0
collected 2 items
test_it.py .. [100%]
============================== 2 passed in 0.01s ===============================
___________________________ test_unraisable_in_setup ___________________________
pytester = <Pytester PosixPath('/tmp/pytest-of-mgorny/pytest-1/test_unraisable_in_setup0')>
@pytest.mark.filterwarnings("default::pytest.PytestUnraisableExceptionWarning")
def test_unraisable_in_setup(pytester: Pytester) -> None:
pytester.makepyfile(
test_it="""
import pytest
class BrokenDel:
def __del__(self):
raise ValueError("del is broken")
@pytest.fixture
def broken_del():
obj = BrokenDel()
del obj
def test_it(broken_del): pass
def test_2(): pass
"""
)
result = pytester.runpytest()
assert result.ret == 0
> assert result.parseoutcomes() == {"passed": 2, "warnings": 1}
E AssertionError: assert {'passed': 2} == {'passed': 2, 'warnings': 1}
E Omitting 1 identical items, use -vv to show
E Right contains 1 more item:
E {'warnings': 1}
E Use -v to get more diff
/tmp/pytest/testing/test_unraisableexception.py:64: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.9.10[pypy-7.3.8-final], pytest-7.2.0.dev12+g22756c28e, pluggy-1.0.0
rootdir: /tmp/pytest-of-mgorny/pytest-1/test_unraisable_in_setup0
collected 2 items
test_it.py .. [100%]
============================== 2 passed in 0.01s ===============================
_________________________ test_unraisable_in_teardown __________________________
pytester = <Pytester PosixPath('/tmp/pytest-of-mgorny/pytest-1/test_unraisable_in_teardown0')>
@pytest.mark.filterwarnings("default::pytest.PytestUnraisableExceptionWarning")
def test_unraisable_in_teardown(pytester: Pytester) -> None:
pytester.makepyfile(
test_it="""
import pytest
class BrokenDel:
def __del__(self):
raise ValueError("del is broken")
@pytest.fixture
def broken_del():
yield
obj = BrokenDel()
del obj
def test_it(broken_del): pass
def test_2(): pass
"""
)
result = pytester.runpytest()
assert result.ret == 0
> assert result.parseoutcomes() == {"passed": 2, "warnings": 1}
E AssertionError: assert {'passed': 2} == {'passed': 2, 'warnings': 1}
E Omitting 1 identical items, use -vv to show
E Right contains 1 more item:
E {'warnings': 1}
E Use -v to get more diff
/tmp/pytest/testing/test_unraisableexception.py:101: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.9.10[pypy-7.3.8-final], pytest-7.2.0.dev12+g22756c28e, pluggy-1.0.0
rootdir: /tmp/pytest-of-mgorny/pytest-1/test_unraisable_in_teardown0
collected 2 items
test_it.py .. [100%]
============================== 2 passed in 0.01s ===============================
________________________ test_unraisable_warning_error _________________________
pytester = <Pytester PosixPath('/tmp/pytest-of-mgorny/pytest-1/test_unraisable_warning_error0')>
@pytest.mark.filterwarnings("error::pytest.PytestUnraisableExceptionWarning")
def test_unraisable_warning_error(pytester: Pytester) -> None:
pytester.makepyfile(
test_it="""
class BrokenDel:
def __del__(self) -> None:
raise ValueError("del is broken")
def test_it() -> None:
obj = BrokenDel()
del obj
def test_2(): pass
"""
)
result = pytester.runpytest()
> assert result.ret == pytest.ExitCode.TESTS_FAILED
E AssertionError: assert <ExitCode.OK: 0> == <ExitCode.TESTS_FAILED: 1>
E + where <ExitCode.OK: 0> = <RunResult ret=ExitCode.OK len(stdout.lines)=8 len(stderr.lines)=0 duration=0.08s>.ret
E + and <ExitCode.TESTS_FAILED: 1> = <enum 'ExitCode'>.TESTS_FAILED
E + where <enum 'ExitCode'> = pytest.ExitCode
/tmp/pytest/testing/test_unraisableexception.py:132: AssertionError
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.9.10[pypy-7.3.8-final], pytest-7.2.0.dev12+g22756c28e, pluggy-1.0.0
rootdir: /tmp/pytest-of-mgorny/pytest-1/test_unraisable_warning_error0
collected 2 items
test_it.py .. [100%]
============================== 2 passed in 0.01s ===============================
_____________________ test_regression_nagative_line_index ______________________
cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x000055851b9b0f20>
when = 'call'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
@classmethod
def from_call(
cls,
func: "Callable[[], TResult]",
when: "Literal['collect', 'setup', 'call', 'teardown']",
reraise: Optional[
Union[Type[BaseException], Tuple[Type[BaseException], ...]]
] = None,
) -> "CallInfo[TResult]":
"""Call func, wrapping the result in a CallInfo.
:param func:
The function to call. Called without arguments.
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
start = timing.time()
precise_start = timing.perf_counter()
try:
> result: Optional[TResult] = func()
/tmp/pytest/.tox/pypy3/lib/pypy3.9/site-packages/_pytest/runner.py:338:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/tmp/pytest/.tox/pypy3/lib/pypy3.9/site-packages/_pytest/runner.py:259: in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
/tmp/pytest/.tox/pypy3/lib/pypy3.9/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
/tmp/pytest/.tox/pypy3/lib/pypy3.9/site-packages/pluggy/_manager.py:80: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
/tmp/pytest/.tox/pypy3/lib/pypy3.9/site-packages/_pytest/unraisableexception.py:88: in pytest_runtest_call
yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
with catch_unraisable_exception() as cm:
yield
if cm.unraisable:
if cm.unraisable.err_msg is not None:
err_msg = cm.unraisable.err_msg
else:
err_msg = "Exception ignored in"
msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
msg += "".join(
traceback.format_exception(
cm.unraisable.exc_type,
cm.unraisable.exc_value,
cm.unraisable.exc_traceback,
)
)
> warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E pytest.PytestUnraisableExceptionWarning: : <function BrokenDel.__del__ at 0x00007f587af42980>
E
E Traceback (most recent call last):
E File "/tmp/pytest-of-mgorny/pytest-1/test_unraisable_warning_error0/test_it.py", line 3, in __del__
E raise ValueError("del is broken")
E ValueError: del is broken
/tmp/pytest/.tox/pypy3/lib/pypy3.9/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning
----------------------------- Captured stdout call -----------------------------
============================= test session starts ==============================
platform linux -- Python 3.9.10[pypy-7.3.8-final], pytest-7.2.0.dev12+g22756c28e, pluggy-1.0.0
rootdir: /tmp/pytest-of-mgorny/pytest-1/test_regression_nagative_line_index0
collected 1 item
test_regression_nagative_line_index.py F [100%]
=================================== FAILURES ===================================
______________________________ test_literal_eval _______________________________
def test_literal_eval():
with pytest.raises(ValueError, match="^$"):
> ast.literal_eval("pytest")
test_regression_nagative_line_index.py:7:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
node_or_string = <_ast.Name object at 0x000055851b2c4420>
def literal_eval(node_or_string):
"""
Safely evaluate an expression node or a string containing a Python
expression. The string or node provided may only consist of the following
Python literal structures: strings, bytes, numbers, tuples, lists, dicts,
sets, booleans, and None.
"""
if isinstance(node_or_string, str):
node_or_string = parse(node_or_string, mode='eval')
if isinstance(node_or_string, Expression):
node_or_string = node_or_string.body
def _raise_malformed_node(node):
raise ValueError(f'malformed node or string: {node!r}')
def _convert_num(node):
if not isinstance(node, Constant) or type(node.value) not in (int, float, complex):
_raise_malformed_node(node)
return node.value
def _convert_signed_num(node):
if isinstance(node, UnaryOp) and isinstance(node.op, (UAdd, USub)):
operand = _convert_num(node.operand)
if isinstance(node.op, UAdd):
return + operand
else:
return - operand
return _convert_num(node)
def _convert(node):
if isinstance(node, Constant):
return node.value
elif isinstance(node, Tuple):
return tuple(map(_convert, node.elts))
elif isinstance(node, List):
return list(map(_convert, node.elts))
elif isinstance(node, Set):
return set(map(_convert, node.elts))
elif (isinstance(node, Call) and isinstance(node.func, Name) and
node.func.id == 'set' and node.args == node.keywords == []):
return set()
elif isinstance(node, Dict):
if len(node.keys) != len(node.values):
_raise_malformed_node(node)
return dict(zip(map(_convert, node.keys),
map(_convert, node.values)))
elif isinstance(node, BinOp) and isinstance(node.op, (Add, Sub)):
left = _convert_signed_num(node.left)
right = _convert_num(node.right)
if isinstance(left, (int, float)) and isinstance(right, complex):
if isinstance(node.op, Add):
return left + right
else:
return left - right
return _convert_signed_num(node)
> return _convert(node_or_string)
/usr/lib/pypy3.9/ast.py:105:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
node = <_ast.Name object at 0x000055851b2c4420>
def _convert(node):
if isinstance(node, Constant):
return node.value
elif isinstance(node, Tuple):
return tuple(map(_convert, node.elts))
elif isinstance(node, List):
return list(map(_convert, node.elts))
elif isinstance(node, Set):
return set(map(_convert, node.elts))
elif (isinstance(node, Call) and isinstance(node.func, Name) and
node.func.id == 'set' and node.args == node.keywords == []):
return set()
elif isinstance(node, Dict):
if len(node.keys) != len(node.values):
_raise_malformed_node(node)
return dict(zip(map(_convert, node.keys),
map(_convert, node.values)))
elif isinstance(node, BinOp) and isinstance(node.op, (Add, Sub)):
left = _convert_signed_num(node.left)
right = _convert_num(node.right)
if isinstance(left, (int, float)) and isinstance(right, complex):
if isinstance(node.op, Add):
return left + right
else:
return left - right
> return _convert_signed_num(node)
/usr/lib/pypy3.9/ast.py:104:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
node = <_ast.Name object at 0x000055851b2c4420>
def _convert_signed_num(node):
if isinstance(node, UnaryOp) and isinstance(node.op, (UAdd, USub)):
operand = _convert_num(node.operand)
if isinstance(node.op, UAdd):
return + operand
else:
return - operand
> return _convert_num(node)
/usr/lib/pypy3.9/ast.py:78:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
node = <_ast.Name object at 0x000055851b2c4420>
def _convert_num(node):
if not isinstance(node, Constant) or type(node.value) not in (int, float, complex):
> _raise_malformed_node(node)
/usr/lib/pypy3.9/ast.py:69:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
node = <_ast.Name object at 0x000055851b2c4420>
def _raise_malformed_node(node):
> raise ValueError(f'malformed node or string: {node!r}')
E ValueError: malformed node or string: <_ast.Name object at 0x000055851b2c4420>
/usr/lib/pypy3.9/ast.py:66: ValueError
During handling of the above exception, another exception occurred:
def test_literal_eval():
with pytest.raises(ValueError, match="^$"):
> ast.literal_eval("pytest")
E AssertionError: Regex pattern '^$' does not match 'malformed node or string: <_ast.Name object at 0x000055851b2c4420>'.
test_regression_nagative_line_index.py:7: AssertionError
=========================== short test summary info ============================
FAILED test_regression_nagative_line_index.py::test_literal_eval - AssertionE...
============================== 1 failed in 0.16s ===============================
=========================== short test summary info ============================
FAILED testing/test_skipping.py::test_errors_in_xfail_skip_expressions - Fail...
FAILED testing/test_unraisableexception.py::test_unraisable - AssertionError:...
FAILED testing/test_unraisableexception.py::test_unraisable_in_setup - Assert...
FAILED testing/test_unraisableexception.py::test_unraisable_in_teardown - Ass...
FAILED testing/test_unraisableexception.py::test_unraisable_warning_error - A...
FAILED testing/code/test_excinfo.py::test_regression_nagative_line_index - py...
====== 6 failed, 3039 passed, 89 skipped, 9 xfailed in 336.42s (0:05:36) =======
pip list
(in .tox/pypy3)
$ pip list
Package Version
------------------ ----------------------
argcomplete 2.0.0
attrs 21.4.0
certifi 2021.10.8
cffi 1.15.0
charset-normalizer 2.0.12
elementpath 2.5.0
greenlet 0.4.13
hpy 0.0.3
hypothesis 6.39.4
idna 3.3
iniconfig 1.1.1
mock 4.0.3
nose 1.3.7
packaging 21.3
pip 22.0.4
pluggy 1.0.0
py 1.11.0
Pygments 2.11.2
pyparsing 3.0.7
pytest 7.2.0.dev12+g22756c28e
readline 6.2.4.1
requests 2.27.1
setuptools 60.9.3
sortedcontainers 2.4.0
tomli 2.0.1
urllib3 1.26.9
wheel 0.37.1
xmlschema 1.10.0
versions
pytest 22756c28e59aaa3e56a2b68732b19e883aa76655
$ pypy3 --version
Python 3.9.10 (8276b505180f70c5784a698a510f0a17317a85c3, Feb 20 2022, 11:21:38)
[PyPy 7.3.8 with GCC 11.2.1 20220115]
Gentoo Linux
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failures with tox -e pypy3 on PyPy 3.9. Start with /tmp/pytest/testing/test_skipping.py and /tmp/pytest/testing/test_unraisableexception.py, focusing on the listed failing tests and their captured output. Done means the affected tests report the expected errors and warnings, including the PytestUnraisableExceptionWarning cases, on the target runtime.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100