raising an exception pytest_runtest_teardown causes all all further tests to fail

Raising an exception in pytest_runtest_teardown in one test will cause an exception in all subsequent tests. This is a regression from 5.2.4 and earlier releases. I use exceptions in pytest_runtest_teardown to assert resource cleanup for all tests regardless of other fixtures that are used. It's unusual to fail in this manner, but this makes it look like all tests are failing.

Here is the output from pytest 6.0.1, where it marks test_a as a failure in the expected fashion. However it marks test_b as a failure with assert colitem in self.stack. If there are more tests, all tests will be marked with the same failure message as test_b.

============================= test session starts ==============================
platform linux -- Python 3.8.2, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/colvin/pytest_repro
collected 2 items

test_exception.py .E.E                                                   [100%]

==================================== ERRORS ====================================
_________________________ ERROR at teardown of test_a __________________________

item = <Function test_a>, nextitem = <Function test_b>

    def pytest_runtest_teardown(item, nextitem):
        global raised
        if not raised:
            raised = True
>           assert False, "fails raising a plain assert"
E           AssertionError: fails raising a plain assert
E           assert False

conftest.py:9: AssertionError
_________________________ ERROR at teardown of test_b __________________________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7f83ab0d4940>
when = 'teardown'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], _T]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: "Optional[Union[Type[BaseException], Tuple[Type[BaseException], ...]]]" = None,
    ) -> "CallInfo[_T]":
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result = func()  # type: Optional[_T]

../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:294:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:247: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
../pytest-venv/lib/python3.8/site-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
../pytest-venv/lib/python3.8/site-packages/pluggy/manager.py:93: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../pytest-venv/lib/python3.8/site-packages/pluggy/manager.py:84: in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:166: in pytest_runtest_teardown
    item.session._setupstate.teardown_exact(item, nextitem)
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:402: in teardown_exact
    self._teardown_towards(needed_collectors)
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:417: in _teardown_towards
    raise exc
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:410: in _teardown_towards
    self._pop_and_teardown()
../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:370: in _pop_and_teardown
    self._teardown_with_finalization(colitem)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <_pytest.runner.SetupState object at 0x7f83ab11fc70>
colitem = <Function test_b>

    def _teardown_with_finalization(self, colitem) -> None:
        self._callfinalizers(colitem)
        colitem.teardown()
        for colitem in self._finalizers:
>           assert colitem in self.stack
E           AssertionError

../pytest-venv/lib/python3.8/site-packages/_pytest/runner.py:391: AssertionError
=========================== short test summary info ============================
ERROR test_exception.py::test_a - AssertionError: fails raising a plain assert
ERROR test_exception.py::test_b - AssertionError
========================= 2 passed, 2 errors in 0.04s ==========================

Here's is the output from pytest 5.4.3

=============================== test session starts ================================
platform linux -- Python 3.8.2, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /home/colvin/pytest_repro
collected 2 items

test_exception.py .E.                                                        [100%]

====================================== ERRORS ======================================
___________________________ ERROR at teardown of test_a ____________________________

item = <Function test_a>, nextitem = <Function test_b>

    def pytest_runtest_teardown(item, nextitem):
        global raised
        if not raised:
            raised = True
>           assert False, "fails raising a plain assert"
E           AssertionError: fails raising a plain assert
E           assert False

conftest.py:9: AssertionError
============================= short test summary info ==============================
ERROR test_exception.py::test_a - AssertionError: fails raising a plain assert
============================ 2 passed, 1 error in 0.01s ============================

To reproduce, I use two files:

conftest.py

import pytest

raised = False

def pytest_runtest_teardown(item, nextitem):
    global raised
    if not raised:
        raised = True
        assert False, "fails raising a plain assert"

test_exception.py

def test_a():
    assert True


def test_b():
    assert True

pip list from python 3.8.2 on ubuntu 20.04

Package        Version
-------------- -------
attrs          19.3.0
iniconfig      1.0.1
more-itertools 8.4.0
packaging      20.4
pip            20.0.2
pkg-resources  0.0.0
pluggy         0.13.1
py             1.9.0
pyparsing      2.4.7
pytest         6.0.1
setuptools     44.0.0
six            1.15.0
toml           0.10.1
wcwidth        0.2.5

Reproduce the regression with the provided conftest.py and test_exception.py, then inspect the teardown path in _pytest/runner.py, especially SetupState.teardown_exact and _teardown_with_finalization. Compare pytest 5.4.3 with 6.0.1; done means the teardown assertion reports test_a while subsequent tests are not incorrectly marked as failures.