RFC. Support for resetting function-scoped fixtures

What's the problem this feature will solve?

Support for per-execution fixtures in executors that wrap each test function and runs the function multiple times (for different examples) have been a long-standing issue, for example:

• https://github.com/pytest-dev/pytest/issues/916
• https://github.com/HypothesisWorks/hypothesis/issues/377

The motivating example is hypothesis, but in an effort to keep this issue more general I'll just refer to it as a "multi-executor" here. I'm not familiar with other multi-executors, but if there are any then the chosen solution should hopefully be usable by those as well.

Some of the reasons that this has been stalled are that the example executions are not up-front enumerable, and that the test item should still be failed/passed and reported as a unit ­— hence, not a good fit for subtests or other declarative-style approaches.

I propose a simpler solution, which is to expose a method that the multi-executor can call in-between examples, to explicitly signal the start of a new example and reset function-scoped fixtures.

Describe the solution you'd like

My proposed solution is, briefly:

1. Add a new scope item, in-between function and class. The new scope is equivalent to function unless running with a multi-executor, but is scoped to the test item so that it is not reset between repeated executions of the function for the same test item. Few fixtures will have the item scope, but an example of a fixture having natural item scope is request.

This new scope is not required for the correctness of the proposed functionality, but since each test item can be executed hundreds of times, there are performance implications that make it highly desirable. (plus, it's a helpful concept for explaining multi-executors in documentation).

2. Add a fixture-reset method to FixtureRequest, to be called by the multi-executor when function-scoped fixtures should be reset.

I realize this is backwards control flow from the usual pytest hooks, but it's only an integration point for multi-executors, not anything that end users will have to care about.

It's also possible to encapsulate the backwards call itself in a pytest-supplied decorator/wrapper — such a wrapper is actually implemented as proof-of-concept in the PR tests, and could be used as an alternative integration point: Execute the wrapper multiple times instead of the original function, and the wrapper takes care of resetting fixtures and passing their new values to the function.

3. To ensure consistent dependency resolution, a sequence counter is added to fixture executions so that they can be reset in the exact same order that they were originally executed.

The rest are implementation details, shown in the (working & with tests) prototype PR #12597.

4. One fixture (tmp_path) was not compatible with resets due to expecting to always having run the end-of-item pytest_runtest_makereport before teardown.

The fix was easy, but it is an indication that there may be other such incompatible fixtures in the wild. I believe this is ok, as they are not impacted unless running with a multi-executor — which was never safe before, and (opt-out) disallowed by at least hypothesis.

What do you think?

Additional context

• https://github.com/pytest-dev/pytest/issues/9699 - requesting a scope below function, to support competing dependency-mutating fixtures. This is different from, and not addressed by, this proposal.
• https://github.com/HypothesisWorks/hypothesis/issues/2370 - providing some examples for the utility of item-scoped fixtures with hypothesis.
• https://github.com/untitaker/pytest-subtesthack/ has been mentioned as an alternative, but it's archived now (and seems to require additional boilerplate instead of being correct-by-default).

Start with prototype PR #12597 and its tests, then read the FixtureRequest integration described in this issue. Check the proposed item scope, fixture reset behavior, execution sequence counter, and the tmp_path interaction with pytest_runtest_makereport. Done means the design is agreed and the behavior is covered by tests for repeated executions and fixture teardown.