dask / dask/distributed

Weakref finalizers prohibit GC

Open
#7,639 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

In a couple of places in our code base we're using weakref.finalizers in a way that ensures that an object is actually never GCed.

The notes specifically state

> Note It is important to ensure that func, args and kwargs do not own any references to obj, either directly or indirectly, since otherwise obj will never be garbage collected. In particular, func should not be a bound method of obj.

see here https://docs.python.org/3/library/weakref.html#weakref.finalize

See a dicussion here https://github.com/dask/distributed/pull/7593#discussion_r1120084558

Almost every usage of weakref.finalize in our codebase is subject to this fallacy. Here a couple of example but this is non exhaustive.

- https://github.com/dask/distributed/blob/8a3b44ce91e363053eb42aef31fecd81d1d167bc/distributed/process.py#L110-L112
- https://github.com/dask/distributed/blob/8a3b44ce91e363053eb42aef31fecd81d1d167bc/distributed/process.py#L148
- https://github.com/dask/distributed/blob/8a3b44ce91e363053eb42aef31fecd81d1d167bc/distributed/comm/asyncio_tcp.py#L474-L477
- https://github.com/dask/distributed/blob/8a3b44ce91e363053eb42aef31fecd81d1d167bc/distributed/comm/inproc.py#L182
- https://github.com/dask/distributed/blob/8a3b44ce91e363053eb42aef31fecd81d1d167bc/distributed/comm/tcp.py#L196-L206

We should remove these broken finalizers since they obviously do not help a lot other than keeping objects alive. Some might have important functionality which is no longer triggered because of the reference.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.