dask / dask/distributed

`pytest`'s`LogCaptureHandler` prevents garbage collection of `Worker` instances (and others)

Open
#6,352 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

In tests that raise an exception, the `LogCaptureHandler` from `pytest` prevents garbage collection by maintaining a reference to the raised exception and its traceback:
![worker_chain](https://user-images.githubusercontent.com/2699097/168824400-360b2dc1-ccb8-4520-a94c-bd2ae1811427.png)

_(This reference chain was created with `objgraph.find_backref_chain` [https://mg.pov.lt/objgraph/#memory-leak-example](https://mg.pov.lt/objgraph/#memory-leak-example))_

Keeping those references alive prevents garbage collection of otherwise released objects and therefore testing for proper garbage collection of `Scheduler`, `TaskState` and `Worker` instances as requested by #6250. For example, in #6344, the failures of `distributed/tests/test_scheduler.py::test_rebalance_workers_and_keys` and several other tests are caused by this.

One possible for the failures in #6344 would be to manually clear the handlers during `check_instances()`, though they seem to be hard to access, which results in a rather brittle solution:
```
handlers = logging.getLogger().handlers
for handler in handlers:
if hasattr(handler, "records"):
handler.records.clear()
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.