pytest-dev / pytest-dev/pytest
Support checking exception order in `RaisesGroup`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.5k
- Forks
- 3.4k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 35
Description
What's the problem this feature will solve?
Some tests need to assert that an exception group was raised with multiple exceptions in a particular order. Currently, making such an assertion about the group must be done somewhat manually, e.g. like
def assert_matches(
exception: BaseException, raises: _pytest.raises.AbstractRaises[BaseException]
) -> None:
assert raises.matches(exception), raises.fail_reason
def check_group(exc_grp: ExceptionGroup[Exception], /) -> bool:
assert len(exc_grp.exceptions) == 2
assert_matches(
exc_grp.exceptions[0],
pytest.RaisesExc(RuntimeError, check=check_body_exc),
)
assert_matches(
exc_grp.exceptions[1],
pytest.RaisesExc(RuntimeError, check=check_aclose_exc),
)
return True
with pytest.raises(ExceptionGroup, check=check_group):
...
or similarly via exc_info:
with pytest.raises(ExceptionGroup) as exc_info:
...
assert many_properties_of(exc_info.value) == ...
Describe the solution you'd like
raises and RaisesGroup have made asserting expected exceptions much more convenient and less mistake-prone than the old
try:
...
except ExpectedExceptionType as exc:
assert some_property(exc) == ...
else:
assert False
and
with pytest.raises(ExceptionGroup) as exc_info:
...
assert enough_properties_about_every_exc_in(exc_info.value) == ...
It would be convenient if RaisesGroup also supported cases where the group has an expected order.
In more detail, the proposal is: add an ordered: bool = True parameter to RaisesGroup. Setting ordered=False would cause RaisesGroup to use a simpler matching algorithm that does not attempt to reorder exceptions and retry AbstractRaises.matches on other children when a .matches on a child fails.
For example, two real-world use cases:
-
We test a class that builds a
BaseExceptionGroupcontaining any exception from the body of itsasync withand an exception raised during a cleanup operation in__aexit__. This assertion was being written aswith pytest.RaisesGroup( pytest.RaisesExc(RuntimeError, check=check_body_exc), pytest.RaisesExc(RuntimeError, check=check_aclose_exc), ): ...until we realized that this wasn't asserting the order and we needed to assert the order explicitly.
-
We test various error, exception, and shutdown semantics of a de/multiplexer class that has a task group containing two service tasks, can propagate an exception from the user (the body of the object's
async with), can raise socket, TLS, and protocol state machine exceptions in the_serve_outgoingand_serve_incomingservice tasks, and can raise another exception during a final shutdown operation in__aexit__after the service task group exits earlier inmultiplexer.__aexit__. There are various test cases that assert thatasync with multiplexer: stuff()raises the correct exceptions in the correct order for that case.
Alternative solutions
It's perfectly possible to make the necessary assertions manually without using RaisesGroup, similar to how expected ExceptionGroups were more often asserted before Trio added RaisesGroup, e.g.
with pytest.raises(BaseExceptionGroup) as exc_info:
...
assert len(exc_info.value.exceptions) == 3
assert isinstance(exc_info.value.exceptions[0], BrokenResourceError)
assert isinstance(exc_info.value.exceptions[0].__cause__, ...)
assert isinstance(exc_info.value.exceptions[1], RuntimeError)
assert property_a(exc_info.value.exceptions[1]) == ...
assert isinstance(exc_info.value.exceptions[2], RuntimeError)
assert property_b(exc_info.value.exceptions[2]) == ...
assert_matches(exc_info.value.exceptions[i], pytest.RaisesExc(...)) can sometimes simplify this a bit, but it's still a bit tedious and too easy to forget to assert the length and/or to forget to make assertions about one or more of the exc_info.value.exceptions, allowing them to slip through. It's a somewhat similar issue to group_contains: https://github.com/pytest-dev/pytest/issues/11538#issuecomment-1914533421.
Aside: while it is true that when using with AbstractRaises, using a shorter body of the with is preferrable (see: flake8-bugbear B908), in cases where the exception group gets raised out of a class's __aexit__ there isn't much of another choice in order to test the behavior. flake8-bugbear won't even yell about with big_AbstractRaises: async with thing: a_bunch_of_stuff(), because there is technically only one top-level statement within big_AbstractRaises's context.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the RaisesGroup implementation and AbstractRaises.matches behavior, focusing on how child exceptions are reordered and retried. Add the ordered parameter described in the proposal, then verify that ordered matching remains the default and that ordered=False checks exceptions in their existing order with corresponding tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100