Adaptive cluster on GP does not add workers

I have a workflow that I ran on an adaptive cluster set to have between 0 and 50 nodes. I ran two dask graphs using this cluster and the first one worked as expected. The cluster scaled to 50 nodes and then back down to 0 nodes before the second started (I was expecting this since there is a good bit that goes on client side before the second graph). After the second started though, no workers were added to the cluster despite seeing ~240k tasks waiting. Here is my session and logs from it:

Session Details

```
# A function here creates a GCPCluster
In [1]: create(0, machine_type='n1-highmem-8', source_image="ukb-gwas-pipeline-nealelab-dask-1607640553", bootstrap=False)
Launching cluster with the following configuration:
Source Image: projects/xxxxxxxxx/global/images/ukb-gwas-pipeline-nealelab-dask-1607640553
Docker Image: eczech/ukb-gwas-pipeline-nealelab:v2.30.0
Machine Type: n1-highmem-8
Filesytsem Size: 50
N-GPU Type:
Zone: us-east1-c
Creating scheduler instance
dask-370d24f7-scheduler
Internal IP: 10.142.0.24
External IP: None
Waiting for scheduler to run
Scheduler is running
/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/contextlib.py:120: UserWarning: Creating your cluster is taking a surprisingly long time. This is likely due to pending resources. Hang tight!
next(self.gen)
Cluster created

In [2]: adapt(0, 50, interval="60s");
distributed.deploy.adaptive - INFO - Adaptive scaling started: minimum=0 maximum=50
Adaptive policy added to cluster (see https://docs.dask.org/en/latest/setup/adaptive.html)

# In another process, the dask job is started

Creating worker instance
dask-370d24f7-worker-b804c1f9
Internal IP: 10.142.0.25
External IP: None
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
Creating worker instance
dask-370d24f7-worker-acccbb3f
Internal IP: 10.142.0.26
External IP: None
dask-370d24f7-worker-64fe0164
Internal IP: 10.142.0.27
External IP: None
dask-370d24f7-worker-7b21f2eb
Internal IP: 10.142.0.29
External IP: None
dask-370d24f7-worker-549dcaba
Internal IP: 10.142.0.30
External IP: None
dask-370d24f7-worker-988564c0
Internal IP: 10.142.0.31
External IP: None
dask-370d24f7-worker-812e97ab
Internal IP: 10.142.0.32
External IP: None
dask-370d24f7-worker-a7d4024d
Internal IP: 10.142.0.33
External IP: None
dask-370d24f7-worker-f0fcca40
Internal IP: 10.142.0.34
External IP: None
dask-370d24f7-worker-030575e4
Internal IP: 10.142.0.35
External IP: None
dask-370d24f7-worker-9c336f06
Internal IP: 10.142.0.36
External IP: None
dask-370d24f7-worker-69e9a59c
Internal IP: 10.142.0.37
External IP: None
dask-370d24f7-worker-2830ff66
Internal IP: 10.142.0.38
External IP: None
dask-370d24f7-worker-c4c6f5c7
Internal IP: 10.142.0.39
External IP: None
dask-370d24f7-worker-5464e88e
Internal IP: 10.142.0.40
External IP: None
dask-370d24f7-worker-6bddb05d
Internal IP: 10.142.0.44
External IP: None
dask-370d24f7-worker-6e0230d5
Internal IP: 10.142.0.48
External IP: None
dask-370d24f7-worker-935727fa
Internal IP: 10.142.0.50
External IP: None
dask-370d24f7-worker-a32d45c9
Internal IP: 10.142.0.57
External IP: None
dask-370d24f7-worker-1ba1daf8
Internal IP: 10.142.0.58
External IP: None
dask-370d24f7-worker-b965acb7
Internal IP: 10.142.0.60
External IP: None
dask-370d24f7-worker-0e50749f
Internal IP: 10.142.0.64
External IP: None
dask-370d24f7-worker-9c5a52ad
Internal IP: 10.142.0.71
External IP: None
dask-370d24f7-worker-d83a7c11
Internal IP: 10.142.0.73
External IP: None
dask-370d24f7-worker-0ac91852
Internal IP: 10.142.0.76
External IP: None
dask-370d24f7-worker-0446ad4a
Internal IP: 10.142.0.79
External IP: None
dask-370d24f7-worker-39859c45
Internal IP: 10.142.0.83
External IP: None
dask-370d24f7-worker-5a7fb3ec
Internal IP: 10.142.0.89
External IP: None
dask-370d24f7-worker-442d049e
Internal IP: 10.142.0.97
External IP: None
dask-370d24f7-worker-2408a595
Internal IP: 10.142.0.104
External IP: None
dask-370d24f7-worker-9fa7042c
Internal IP: 10.142.15.199
External IP: None
dask-370d24f7-worker-2cefd81e
Internal IP: 10.142.15.200
External IP: None
dask-370d24f7-worker-35c81586
Internal IP: 10.142.15.201
External IP: None
dask-370d24f7-worker-6c6802aa
Internal IP: 10.142.15.202
External IP: None
dask-370d24f7-worker-f1a8ca6b
Internal IP: 10.142.15.203
External IP: None
dask-370d24f7-worker-3f60cbde
Internal IP: 10.142.15.204
External IP: None
dask-370d24f7-worker-20acd644
Internal IP: 10.142.15.205
External IP: None
dask-370d24f7-worker-e35908d2
Internal IP: 10.142.15.206
External IP: None
dask-370d24f7-worker-1228a2ca
Internal IP: 10.142.15.207
External IP: None
dask-370d24f7-worker-65f49f2a
Internal IP: 10.142.15.208
External IP: None
dask-370d24f7-worker-dc8cc315
Internal IP: 10.142.15.209
External IP: None
dask-370d24f7-worker-614d3162
Internal IP: 10.142.15.210
External IP: None
dask-370d24f7-worker-ccec48b4
Internal IP: 10.142.15.211
External IP: None
dask-370d24f7-worker-49465c77
Internal IP: 10.142.15.212
External IP: None
dask-370d24f7-worker-4b13cb9e
Internal IP: 10.142.15.213
External IP: None
dask-370d24f7-worker-758d8b4e
Internal IP: 10.142.15.214
External IP: None
dask-370d24f7-worker-456f5a1f
Internal IP: 10.142.15.215
External IP: None
dask-370d24f7-worker-17c83e64
Internal IP: 10.142.15.216
External IP: None
dask-370d24f7-worker-ffd27a35
Internal IP: 10.142.15.217
External IP: None
dask-370d24f7-worker-15953533
Internal IP: 10.142.15.218
External IP: None
In [3]: distributed.deploy.adaptive - INFO - Retiring workers [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]
Closing Instance: dask-370d24f7-worker-5a7fb3ec
Closing Instance: dask-370d24f7-worker-9c5a52ad
Closing Instance: dask-370d24f7-worker-988564c0
Closing Instance: dask-370d24f7-worker-1ba1daf8
Closing Instance: dask-370d24f7-worker-e35908d2
Closing Instance: dask-370d24f7-worker-456f5a1f
Closing Instance: dask-370d24f7-worker-64fe0164
Closing Instance: dask-370d24f7-worker-2830ff66
Closing Instance: dask-370d24f7-worker-2cefd81e
Closing Instance: dask-370d24f7-worker-614d3162
Closing Instance: dask-370d24f7-worker-ffd27a35
Closing Instance: dask-370d24f7-worker-15953533
Closing Instance: dask-370d24f7-worker-f0fcca40
Closing Instance: dask-370d24f7-worker-0ac91852
Closing Instance: dask-370d24f7-worker-812e97ab
Closing Instance: dask-370d24f7-worker-17c83e64
Closing Instance: dask-370d24f7-worker-f1a8ca6b
Closing Instance: dask-370d24f7-worker-acccbb3f
Closing Instance: dask-370d24f7-worker-b804c1f9
Closing Instance: dask-370d24f7-worker-0e50749f
Closing Instance: dask-370d24f7-worker-935727fa
Closing Instance: dask-370d24f7-worker-0446ad4a
Closing Instance: dask-370d24f7-worker-7b21f2eb
Closing Instance: dask-370d24f7-worker-5464e88e
In [3]: distributed.deploy.adaptive - INFO - Retiring workers [24, 25, 26, 27]
Closing Instance: dask-370d24f7-worker-49465c77
Closing Instance: dask-370d24f7-worker-1228a2ca
Closing Instance: dask-370d24f7-worker-69e9a59c
Closing Instance: dask-370d24f7-worker-65f49f2a
In [3]: distributed.deploy.adaptive - INFO - Retiring workers [28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]
Closing Instance: dask-370d24f7-worker-6bddb05d
Closing Instance: dask-370d24f7-worker-442d049e
Closing Instance: dask-370d24f7-worker-9fa7042c
Closing Instance: dask-370d24f7-worker-2408a595
Closing Instance: dask-370d24f7-worker-b965acb7
Closing Instance: dask-370d24f7-worker-a7d4024d
Closing Instance: dask-370d24f7-worker-549dcaba
Closing Instance: dask-370d24f7-worker-20acd644
Closing Instance: dask-370d24f7-worker-ccec48b4
Closing Instance: dask-370d24f7-worker-6c6802aa
Closing Instance: dask-370d24f7-worker-39859c45
Closing Instance: dask-370d24f7-worker-4b13cb9e
Closing Instance: dask-370d24f7-worker-6e0230d5
Closing Instance: dask-370d24f7-worker-3f60cbde
Closing Instance: dask-370d24f7-worker-c4c6f5c7
Closing Instance: dask-370d24f7-worker-d83a7c11
Closing Instance: dask-370d24f7-worker-030575e4
Closing Instance: dask-370d24f7-worker-758d8b4e
Closing Instance: dask-370d24f7-worker-a32d45c9
Closing Instance: dask-370d24f7-worker-9c336f06
In [3]: distributed.deploy.adaptive - INFO - Retiring workers [48, 49]
Closing Instance: dask-370d24f7-worker-35c81586
Closing Instance: dask-370d24f7-worker-dc8cc315
Creating worker instance
dask-370d24f7-worker-b7d42149
Internal IP: 10.142.0.3
External IP: None
In [3]: distributed.deploy.adaptive - INFO - Retiring workers [49]
Closing Instance: dask-370d24f7-worker-b7d42149
Closing Instance: dask-370d24f7-worker-b7d42149
Creating worker instance
Task exception was never retrieved
future: exception=BrokenPipeError(32, 'Broken pipe')>
Traceback (most recent call last):
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/asyncio/tasks.py", line 692, in _wrap_awaitable
return (yield from awaitable.__await__())
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/distributed/deploy/spec.py", line 71, in _
await self.start()
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/dask_cloudprovider/gcp/instances.py", line 335, in start
await self.start_worker()
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/dask_cloudprovider/gcp/instances.py", line 339, in start_worker
self.internal_ip, self.external_ip = await self.create_vm()
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/dask_cloudprovider/gcp/instances.py", line 193, in create_vm
self.cluster.compute.instances()
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/googleapiclient/http.py", line 900, in execute
resp, content = _retry_request(
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/googleapiclient/http.py", line 204, in _retry_request
raise exception
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/googleapiclient/http.py", line 177, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/google_auth_httplib2.py", line 189, in request
self.credentials.before_request(
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/google/auth/credentials.py", line 133, in before_request
self.refresh(request)
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/google/oauth2/credentials.py", line 200, in refresh
access_token, refresh_token, expiry, grant_response = _client.refresh_grant(
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/google/oauth2/_client.py", line 248, in refresh_grant
response_data = _token_endpoint_request(request, token_uri, body)
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/google/oauth2/_client.py", line 105, in _token_endpoint_request
response = request(method="POST", url=token_uri, headers=headers, body=body)
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/google_auth_httplib2.py", line 116, in __call__
response, data = self.http.request(
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/httplib2/__init__.py", line 1985, in request
(response, content) = self._request(
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/httplib2/__init__.py", line 1650, in _request
(response, content) = self._conn_request(
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/httplib2/__init__.py", line 1558, in _conn_request
conn.request(method, request_uri, body, headers)
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/http/client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/http/client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/http/client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/http/client.py", line 1049, in _send_output
self.send(chunk)
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/http/client.py", line 971, in send
self.sock.sendall(data)
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/ssl.py", line 1204, in sendall
v = self.send(byte_view[count:])
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/ssl.py", line 1173, in send
return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe
In [3]: distributed.deploy.adaptive - INFO - Retiring workers [49]
Closing Instance: dask-370d24f7-worker-cfe9aa6c
Task exception was never retrieved
future: exception=>
Traceback (most recent call last):
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/dask_cloudprovider/gcp/instances.py", line 254, in close
self.cluster.compute.instances().delete(
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/eczech/miniconda3/envs/cloudprovider/lib/python3.8/site-packages/googleapiclient/http.py", line 915, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError:
```

The only evidence I have that tasks are waiting is the UI, which looks like this:


Let me know if there is some other information that would be useful in these situations.

I am working from the code in https://github.com/dask/dask-cloudprovider/commit/ad4eb0eb7444c35ce3beaedc4c510abe6706f03e (latest master commit at TOW).

**Update**: After waiting a little longer, there is a stack trace that dumped out. It looks like an attempt was made to retire one of the nodes twice. This has been idling for ~2.5 hours now though so should it be waiting so long between attempts or is there anything else that might block the code for that much time?