`tornado`'s `call_later`, `add_callback`, etc. prevent garbage collection of otherwise released objects
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
As mentioned in [https://github.com/dask/distributed/pull/6246/files#r861616968](https://github.com/dask/distributed/pull/6246/files#r861616968), `call_later` and `add_callback` maintain references unless cancelled. We do not properly cancel callbacks in particular when closing `Scheduler`, `Nanny` or `Worker` instances. See below for an example:

_(This reference chain was created with `objgraph.find_backref_chain` [https://mg.pov.lt/objgraph/#memory-leak-example](https://mg.pov.lt/objgraph/#memory-leak-example))_
This in turn prevents garbage collection of otherwise released objects, which in turn prevents testing for proper garbage collection as requested in #6250. One possible solution has been sketched out in #6345, namely maintaining a registry of active callbacks, which can be used to cancel all active callbacks on close.
Instead of implementing this solution in the various places where it is needed, I'd suggest moving away from `tornado` while we are adjusting the callbacks and relying on `asyncio.Task`s that we store in a background task registry introduced and cleaned on `ServerNode` or `Server`.
Contributor guide
Assessment
This issue has not been assessed yet.