excessive CPU time spent on gc (even after manually adjusting gc thresholds)
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
During the process I'm running I very quickly get this warning (for pretty much each worker)
```
distributed.utils_perf - WARNING - full garbage collections took 36% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 34% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 35% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 34% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 35% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 35% CPU time recently (threshold: 10%)
```
However, the percent of memory used is by each worker is low -

I have plenty of memory to work with so my question is - is it possible dask is aggressively doing gc? If so is it possible to change the gc threshold/collect less aggressively?
I've tried to manually adjust the `gc.threshold` but this seems to have no effect
```
g0, g1, g2 = gc.get_threshold()
gc.set_threshold(g0*10, g1*10, g2*10)
```
(using distributed 1.28.1 and dask 1.2.2)
Contributor guide
Assessment
This issue has not been assessed yet.