dask / dask/distributed

Possible lead for `WARNING - full garbage collections took XY% CPU time recently` on CI

Open
#6,080 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

https://github.com/dask/distributed/runs/5820320487?check_suite_focus=true

In the above linked test job I have another case of spamming `full garbage collection took XY% CPU time` warnings up to 99% until the job actually fails.

The interesting bit is now in the timestamps we've introduces recently. Namely, the point where an exception is raised up to the point where the test actually breaks off is more or less exactly one second (modulo a few ms).

Searching for timeouts of exactly one second, there are a few possible candidates around, e.g.

- [`check_active_rpc`](https://github.com/dask/distributed/blob/a610f73ea2beea3715d08f6d4b70f930da0b76fd/distributed/utils_test.py#L511-L533)
- [`threadpoolexecutor._worker`](https://github.com/dask/distributed/blob/a610f73ea2beea3715d08f6d4b70f930da0b76fd/distributed/threadpoolexecutor.py#L51)
- [`utils_test::start_cluster`](https://github.com/dask/distributed/blob/a610f73ea2beea3715d08f6d4b70f930da0b76fd/distributed/utils_test.py#L866)

Of these the most likley culprit is `check_active_rpc`. It calls into `async_wait_for`

https://github.com/dask/distributed/blob/a610f73ea2beea3715d08f6d4b70f930da0b76fd/distributed/utils_test.py#L1239-L1246

which appears to run a rather tight loop by rechecking the condition every `ms`.

---

Truncated logs
```
2022-04-04 16:56:17,606 - distributed.worker - WARNING - Heartbeat to scheduler failed
...
2022-04-04 16:56:17,610 - distributed.scheduler - INFO - Scheduler closing...
2022-04-04 16:56:17,611 - distributed.scheduler - INFO - Scheduler closing all comms
2022-04-04 16:56:17,833 - distributed.utils_perf - WARNING - full garbage collections took 33% CPU time recently (threshold: 10%)
...
2022-04-04 16:56:22,619 - distributed.utils_perf - WARNING - full garbage collections took 40% CPU time recently (threshold: 10%)
2022-04-04 16:56:22,620 - distributed.comm.tcp - INFO - Connection from tcp://127.0.0.1:57850 closed before handshake completed
2022-04-04 16:56:22,620 - distributed.comm.tcp - INFO - Connection from tcp://127.0.0.1:57852 closed before handshake completed
2022-04-04 16:56:22,867 - distributed.utils_perf - WARNING - full garbage collections took 40% CPU time recently (threshold: 10%)
...

2022-04-04 16:57:16,469 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
=============================== warnings summary ===============================
...
```

Full log

```
2022-04-04 16:56:16,485 - distributed.scheduler - INFO - Clear task state
2022-04-04 16:56:16,486 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:45831
2022-04-04 16:56:16,486 - distributed.scheduler - INFO - dashboard at: 127.0.0.1:33319
2022-04-04 16:56:16,493 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:41141
2022-04-04 16:56:16,493 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:41141
2022-04-04 16:56:16,493 - distributed.worker - INFO - dashboard at: 127.0.0.1:41775
2022-04-04 16:56:16,493 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:45831
2022-04-04 16:56:16,493 - distributed.worker - INFO - -------------------------------------------------
2022-04-04 16:56:16,493 - distributed.worker - INFO - Threads: 1
2022-04-04 16:56:16,493 - distributed.worker - INFO - Memory: 6.78 GiB
2022-04-04 16:56:16,493 - distributed.worker - INFO - Local Directory: /home/runner/work/distributed/distributed/dask-worker-space/worker-zge53d2k
2022-04-04 16:56:16,493 - distributed.worker - INFO - -------------------------------------------------
2022-04-04 16:56:16,494 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:42423
2022-04-04 16:56:16,494 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:42423
2022-04-04 16:56:16,494 - distributed.worker - INFO - dashboard at: 127.0.0.1:45093
2022-04-04 16:56:16,494 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:45831
2022-04-04 16:56:16,495 - distributed.worker - INFO - -------------------------------------------------
2022-04-04 16:56:16,495 - distributed.worker - INFO - Threads: 2
2022-04-04 16:56:16,495 - distributed.worker - INFO - Memory: 6.78 GiB
2022-04-04 16:56:16,495 - distributed.worker - INFO - Local Directory: /home/runner/work/distributed/distributed/dask-worker-space/worker-pieu6kp5
2022-04-04 16:56:16,495 - distributed.worker - INFO - -------------------------------------------------
2022-04-04 16:56:16,547 - distributed.scheduler - INFO - Register worker
2022-04-04 16:56:16,570 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:41141
2022-04-04 16:56:16,571 - distributed.core - INFO - Starting established connection
2022-04-04 16:56:16,571 - distributed.scheduler - INFO - Register worker
2022-04-04 16:56:16,593 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:42423
2022-04-04 16:56:16,593 - distributed.core - INFO - Starting established connection
2022-04-04 16:56:16,594 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:45831
2022-04-04 16:56:16,594 - distributed.worker - INFO - -------------------------------------------------
2022-04-04 16:56:16,594 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:45831
2022-04-04 16:56:16,595 - distributed.worker - INFO - -------------------------------------------------
2022-04-04 16:56:16,595 - distributed.core - INFO - Starting established connection
2022-04-04 16:56:16,595 - distributed.core - INFO - Starting established connection
2022-04-04 16:56:16,635 - distributed.scheduler - INFO - Receive client connection: Client-248d82d1-b438-11ec-89ea-000d3aef0b2d
2022-04-04 16:56:16,657 - distributed.core - INFO - Starting established connection
2022-04-04 16:56:17,588 - distributed.scheduler - INFO - Remove client Client-248d82d1-b438-11ec-89ea-000d3aef0b2d
2022-04-04 16:56:17,590 - distributed.scheduler - INFO - Remove client Client-248d82d1-b438-11ec-89ea-000d3aef0b2d
2022-04-04 16:56:17,594 - distributed.scheduler - INFO - Close client connection: Client-248d82d1-b438-11ec-89ea-000d3aef0b2d
2022-04-04 16:56:17,599 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:41141
2022-04-04 16:56:17,599 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:42423
2022-04-04 16:56:17,602 - distributed.scheduler - INFO - Remove worker
2022-04-04 16:56:17,603 - distributed.core - INFO - Removing comms to tcp://127.0.0.1:41141
2022-04-04 16:56:17,603 - distributed.scheduler - INFO - Remove worker
2022-04-04 16:56:17,603 - distributed.core - INFO - Removing comms to tcp://127.0.0.1:42423
2022-04-04 16:56:17,603 - distributed.scheduler - INFO - Lost all workers
2022-04-04 16:56:17,606 - distributed.worker - WARNING - Heartbeat to scheduler failed
Traceback (most recent call last):
File "/home/runner/work/distributed/distributed/distributed/core.py", line 1083, in _connect
comm = await connect(
File "/home/runner/work/distributed/distributed/distributed/comm/core.py", line 301, in connect
await waiter.wait()
File "/usr/share/miniconda3/envs/dask-distributed/lib/python3.10/asyncio/locks.py", line 214, in wait
await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/runner/work/distributed/distributed/distributed/worker.py", line 1129, in heartbeat
response = await retry_operation(
File "/home/runner/work/distributed/distributed/distributed/utils_comm.py", line 386, in retry_operation
return await retry(
File "/home/runner/work/distributed/distributed/distributed/utils_comm.py", line 371, in retry
return await coro()
File "/home/runner/work/distributed/distributed/distributed/core.py", line 937, in send_recv_from_rpc
comm = await self.pool.connect(self.addr)
File "/home/runner/work/distributed/distributed/distributed/core.py", line 1147, in connect
return await connect_attempt
File "/home/runner/work/distributed/distributed/distributed/core.py", line 1103, in _connect
raise CommClosedError("ConnectionPool closing.")
distributed.comm.core.CommClosedError: ConnectionPool closing.
2022-04-04 16:56:17,606 - distributed.worker - WARNING - Heartbeat to scheduler failed
Traceback (most recent call last):
File "/home/runner/work/distributed/distributed/distributed/core.py", line 1083, in _connect
comm = await connect(
File "/home/runner/work/distributed/distributed/distributed/comm/core.py", line 301, in connect
await waiter.wait()
File "/usr/share/miniconda3/envs/dask-distributed/lib/python3.10/asyncio/locks.py", line 214, in wait
await fut
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/runner/work/distributed/distributed/distributed/worker.py", line 1129, in heartbeat
response = await retry_operation(
File "/home/runner/work/distributed/distributed/distributed/utils_comm.py", line 386, in retry_operation
return await retry(
File "/home/runner/work/distributed/distributed/distributed/utils_comm.py", line 371, in retry
return await coro()
File "/home/runner/work/distributed/distributed/distributed/core.py", line 937, in send_recv_from_rpc
comm = await self.pool.connect(self.addr)
File "/home/runner/work/distributed/distributed/distributed/core.py", line 1147, in connect
return await connect_attempt
File "/home/runner/work/distributed/distributed/distributed/core.py", line 1103, in _connect
raise CommClosedError("ConnectionPool closing.")
distributed.comm.core.CommClosedError: ConnectionPool closing.
2022-04-04 16:56:17,610 - distributed.scheduler - INFO - Scheduler closing...
2022-04-04 16:56:17,611 - distributed.scheduler - INFO - Scheduler closing all comms
2022-04-04 16:56:17,833 - distributed.utils_perf - WARNING - full garbage collections took 33% CPU time recently (threshold: 10%)
2022-04-04 16:56:18,079 - distributed.utils_perf - WARNING - full garbage collections took 34% CPU time recently (threshold: 10%)
2022-04-04 16:56:18,313 - distributed.utils_perf - WARNING - full garbage collections took 34% CPU time recently (threshold: 10%)
2022-04-04 16:56:18,557 - distributed.utils_perf - WARNING - full garbage collections took 34% CPU time recently (threshold: 10%)
2022-04-04 16:56:18,811 - distributed.utils_perf - WARNING - full garbage collections took 35% CPU time recently (threshold: 10%)
2022-04-04 16:56:19,064 - distributed.utils_perf - WARNING - full garbage collections took 35% CPU time recently (threshold: 10%)
2022-04-04 16:56:19,315 - distributed.utils_perf - WARNING - full garbage collections took 36% CPU time recently (threshold: 10%)
2022-04-04 16:56:19,571 - distributed.utils_perf - WARNING - full garbage collections took 36% CPU time recently (threshold: 10%)
2022-04-04 16:56:19,824 - distributed.utils_perf - WARNING - full garbage collections took 37% CPU time recently (threshold: 10%)
2022-04-04 16:56:20,079 - distributed.utils_perf - WARNING - full garbage collections took 37% CPU time recently (threshold: 10%)
2022-04-04 16:56:20,332 - distributed.utils_perf - WARNING - full garbage collections took 37% CPU time recently (threshold: 10%)
2022-04-04 16:56:20,587 - distributed.utils_perf - WARNING - full garbage collections took 38% CPU time recently (threshold: 10%)
2022-04-04 16:56:20,842 - distributed.utils_perf - WARNING - full garbage collections took 38% CPU time recently (threshold: 10%)
2022-04-04 16:56:21,095 - distributed.utils_perf - WARNING - full garbage collections took 38% CPU time recently (threshold: 10%)
2022-04-04 16:56:21,351 - distributed.utils_perf - WARNING - full garbage collections took 38% CPU time recently (threshold: 10%)
2022-04-04 16:56:21,606 - distributed.utils_perf - WARNING - full garbage collections took 38% CPU time recently (threshold: 10%)
2022-04-04 16:56:21,858 - distributed.utils_perf - WARNING - full garbage collections took 39% CPU time recently (threshold: 10%)
2022-04-04 16:56:22,114 - distributed.utils_perf - WARNING - full garbage collections took 39% CPU time recently (threshold: 10%)
2022-04-04 16:56:22,363 - distributed.utils_perf - WARNING - full garbage collections took 40% CPU time recently (threshold: 10%)
2022-04-04 16:56:22,619 - distributed.utils_perf - WARNING - full garbage collections took 40% CPU time recently (threshold: 10%)
2022-04-04 16:56:22,620 - distributed.comm.tcp - INFO - Connection from tcp://127.0.0.1:57850 closed before handshake completed
2022-04-04 16:56:22,620 - distributed.comm.tcp - INFO - Connection from tcp://127.0.0.1:57852 closed before handshake completed
2022-04-04 16:56:22,867 - distributed.utils_perf - WARNING - full garbage collections took 40% CPU time recently (threshold: 10%)
2022-04-04 16:56:23,121 - distributed.utils_perf - WARNING - full garbage collections took 41% CPU time recently (threshold: 10%)
2022-04-04 16:56:23,368 - distributed.utils_perf - WARNING - full garbage collections took 42% CPU time recently (threshold: 10%)
2022-04-04 16:56:23,614 - distributed.utils_perf - WARNING - full garbage collections took 43% CPU time recently (threshold: 10%)
2022-04-04 16:56:23,868 - distributed.utils_perf - WARNING - full garbage collections took 44% CPU time recently (threshold: 10%)
2022-04-04 16:56:24,121 - distributed.utils_perf - WARNING - full garbage collections took 44% CPU time recently (threshold: 10%)
2022-04-04 16:56:24,377 - distributed.utils_perf - WARNING - full garbage collections took 54% CPU time recently (threshold: 10%)
2022-04-04 16:56:24,629 - distributed.utils_perf - WARNING - full garbage collections took 57% CPU time recently (threshold: 10%)
2022-04-04 16:56:24,886 - distributed.utils_perf - WARNING - full garbage collections took 73% CPU time recently (threshold: 10%)
2022-04-04 16:56:25,143 - distributed.utils_perf - WARNING - full garbage collections took 86% CPU time recently (threshold: 10%)
2022-04-04 16:56:25,399 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:25,655 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:25,913 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:26,171 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:26,427 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:26,681 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:26,931 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:27,182 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:27,436 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:27,691 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:27,949 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:28,206 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:28,459 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:28,712 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:28,970 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:29,226 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:29,480 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:29,737 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:29,990 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:30,250 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:30,504 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:30,753 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:31,003 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:31,256 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:31,513 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:31,770 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:32,031 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:32,293 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:32,553 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:32,814 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:33,076 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:33,336 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:33,598 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:33,861 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:34,113 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:34,367 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:34,622 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:34,876 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:35,131 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:35,391 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:35,649 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:35,907 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:36,164 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:36,418 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:36,675 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:36,939 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:37,202 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:37,463 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:37,719 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:37,976 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:38,235 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:38,492 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:38,751 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:38,997 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:39,246 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:39,493 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:39,756 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:40,011 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:40,266 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:40,527 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:40,786 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:41,044 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:41,301 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:41,559 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:41,820 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:42,079 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:42,336 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:42,592 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:42,848 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:43,111 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:43,365 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:43,628 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:43,882 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:44,145 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:44,409 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:44,672 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:44,934 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:45,197 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:45,458 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:45,718 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:45,979 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:46,238 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:46,497 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:46,756 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:47,018 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:47,279 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:47,540 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:47,796 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:48,056 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:48,315 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:48,571 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:48,825 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:49,077 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:49,337 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:49,597 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:49,857 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:50,118 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:50,378 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:50,635 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:50,892 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:51,151 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:51,415 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:51,669 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:51,924 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:52,173 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:52,430 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:52,682 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:52,940 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:53,194 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:53,448 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:53,709 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:53,973 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:54,242 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:54,513 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:54,784 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:55,055 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:55,326 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:55,596 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:55,866 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:56,136 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:56,407 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:56,678 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:56,948 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:57,218 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:57,492 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:57,765 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:58,038 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:58,311 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:58,584 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:58,857 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:59,128 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:59,400 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:59,672 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:56:59,943 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:00,209 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:00,470 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:00,726 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:00,990 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:01,251 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:01,510 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:01,766 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:02,023 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:02,281 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:02,544 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:02,796 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:03,057 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:03,321 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:03,584 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:03,845 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:04,109 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:04,368 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:04,631 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:04,892 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:05,155 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:05,416 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:05,676 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:05,933 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:06,197 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:06,448 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:06,709 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:06,967 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:07,225 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:07,481 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:07,737 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:07,995 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:08,257 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:08,517 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:08,774 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:09,029 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:09,290 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:09,540 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:09,785 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:10,031 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:10,285 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:10,541 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:10,798 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:11,055 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:11,315 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:11,572 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:11,830 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:12,089 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:12,347 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:12,603 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:12,863 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:13,125 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:13,379 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:13,634 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:13,890 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:14,145 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:14,399 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:14,656 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:14,912 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:15,169 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:15,432 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:15,692 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:15,950 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:16,210 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
2022-04-04 16:57:16,469 - distributed.utils_perf - WARNING - full garbage collections took 99% CPU time recently (threshold: 10%)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.