Reference leaks caused by `@pytest.mark.parametrize` on newer Python versions

The combination of PyTest, and @pytest.mark.parametrize causes reference leaks: by this, I mean that the objects parameterizing the test case are not always reliably freed by the time the Python interpreter has shut down. The behavior is somewhat erratic appears most noticeable with the latest Python 3.12.

Here is a basic example involving pure Python code:

import pytest

class Foo:
    def __init__(self, value):
        self.value = value
        print(f'created {self.value}')

    def __del__(self):
        print(f'deleted {self.value}')

i1 = Foo(1)
i2 = Foo(2)

@pytest.mark.parametrize('i', [i1])
def test_foo(i):
    pass

With this, I get (on Python 3.8):

$ python3.8 -m pytest foo.py --capture no
====================================== test session starts ======================================
platform darwin -- Python 3.8.18, pytest-7.4.4, pluggy-1.0.0
rootdir: /Users/wjakob
collecting ... created 1
created 2
collected 1 item

foo.py .

======================================= 1 passed in 0.00s =======================================
deleted 1
deleted 2

On Python 3.12, I get

$ python3.12 -m pytest foo.py --capture no
====================================== test session starts ======================================
platform darwin -- Python 3.12.1, pytest-7.4.4, pluggy-1.3.0
rootdir: /Users/wjakob
plugins: anyio-4.2.0
collecting ... created 1
created 2
collected 1 item

foo.py .

======================================= 1 passed in 0.00s =======================================
deleted 2

In other words, the Foo(1) instance passed to @pytest.mark.parametrize never had their __del__ method called.

One remark right away: the use of the _del__ method is of course considered bad practice in Python. I only used it to make a truly minimal example.

The larger context is as follows: I'm the author of the nanobind C++ <-> Python generator and co-author of pybind11. I want these tools to report object leaks in Python bindings, which can turn into a quite serious problem when the tooling provides no hints about such leaks taking place.

The problem is that C++ projects with bindings that use pytest in their test suite now report leaks that aren't the fault of these extensions but due to something weird happening with PyTest, specifically on newer Python versions.

There seems to be some issue related to how PyTest stores the @pytest.mark.parametrize information that prevents Python's cyclic GC from being able to collect it before the interpreter shuts down.

Start with the minimal Python reproducer in the issue and inspect how pytest stores @pytest.mark.parametrize data during collection and teardown. Compare behavior across the reported Python versions; done means the parameterized Foo instance is released before interpreter shutdown without breaking parameterized tests.