Pytest keeps reference to attributes of failed test cases for too long

Hi,
While running some tests with large numpy arrays, I figured out that there is a difference in how pytest cleans up the data for the failed and passed test cases.
For the failed tests, the references to local variables are kept, leading to the accumulation of many large arrays in the memory and eventually out-memory error.

It could probably be best explained with examples

#test.py
import numpy as np
N = 20_000_000

def get_data():
    d = np.random.rand(N,100)
    return d

def process_data(d):
    return np.diff(d)

def test1():
    d = get_data()
    process_data(d)
    #assert False
    
def test2():
    d = get_data()
    process_data(d)

I also logged memory consumption using the following pytest fixture

#conftest.py
@pytest.fixture(autouse=True)
def record_memory_consumption():
    process = psutil.Process()
    mem_use = process.memory_info().rss
    print(f"Memory consumption: {mem_use / 1024 / 1024:.2f} MB")
    #objgraph.show_growth(limit=3)
    yield 
    mem_use = process.memory_info().rss
    print(f"Memory consumption after test done: {mem_use / 1024 / 1024:.2f} MB")
    #objgraph.show_growth()

And here is the result. Nothing is unexpected.

But if one test fails, the second one will fail too as there is no longer enough memory in the python process.

#test.py
import numpy as np
N = 20_000_000

def get_data():
    d = np.random.rand(N,100)
    return d

def process_data(d):
    return np.diff(d)

def test1():
    d = get_data()
    process_data(d)
    assert False  #simulate failure
    
def test2():
    d = get_data()
    process_data(d)

I used objgraph to trace the reference to the array. It looks like a frame used by the traceback hold onto it.

I think this could be a bug in pytest and may be there is a known work around for this issue.

Pytest 8.1.1
Python 3.11.0
OS: Ubuntu 22.04.3 LTS
Install Packages
Package Version
iniconfig 2.0.0
numpy 1.26.4
packaging 24.0
pip 23.3.1
pluggy 1.4.0
psutil 5.9.8
pytest 8.1.1
setuptools 68.2.2
wheel 0.41.2

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

Start by running the supplied test.py and conftest.py examples with pytest 8.1.1, comparing memory after passed and failed tests. Trace the failing test's retained traceback frame and identify where pytest keeps it; done means large local arrays are released after failure and the behavior is covered by a regression test.