dask / dask/dask-cloudprovider

RuntimeError: cannot schedule new futures after shutdown thrown during cluster shutdown

Open
#359 1 comment 5 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

**What happened**:
RuntimeErrors are thrown at the end of execution when the cluster is being killed. As a result, cloud resources such as TaskDefinitionLists and IAMRole seem to linger on.
**What you expected to happen**:
No errors thrown during the shutdown procedure. I tested the same minimal example in a py3.8 env and could not reproduce this issue; however, I could reproduce the same issue in a py3.9 environment.

**Environment**:

- Dask version: 2022.6.1
- Python version: 3.10.5
- Operating System: macOS 12.4 (21F79)
- Install method (conda, pip, source): conda

Py3.10 environment:
```sh
name: test
channels:
- conda-forge
dependencies:
- aiobotocore=2.3.4=pyhd8ed1ab_0
- aiohttp=3.8.1=py310h1961e1f_1
- aioitertools=0.10.0=pyhd8ed1ab_0
- aiosignal=1.2.0=pyhd8ed1ab_0
- async-timeout=4.0.2=pyhd8ed1ab_0
- attrs=21.4.0=pyhd8ed1ab_0
- bokeh=2.4.3=py310h2ec42d9_0
- botocore=1.24.21=pyhd8ed1ab_1
- brotlipy=0.7.0=py310h1961e1f_1004
- bzip2=1.0.8=h0d85af4_4
- ca-certificates=2022.6.15=h033912b_0
- certifi=2022.6.15=py310h2ec42d9_0
- cffi=1.15.1=py310h96bbf6e_0
- charset-normalizer=2.1.0=pyhd8ed1ab_0
- click=8.1.3=py310h2ec42d9_0
- cloudpickle=2.1.0=pyhd8ed1ab_0
- cryptography=37.0.2=py310h52c3658_0
- cytoolz=0.11.2=py310h1961e1f_2
- dask=2022.6.1=pyhd8ed1ab_0
- dask-cloudprovider=2022.5.0=pyhd8ed1ab_0
- dask-core=2022.6.1=pyhd8ed1ab_0
- distributed=2022.6.1=pyhd8ed1ab_0
- freetype=2.10.4=h4cff582_1
- frozenlist=1.3.0=py310h1961e1f_1
- fsspec=2022.5.0=pyhd8ed1ab_0
- giflib=5.2.1=hbcb3906_2
- heapdict=1.0.1=py_0
- idna=3.3=pyhd8ed1ab_0
- jinja2=3.1.2=pyhd8ed1ab_1
- jmespath=1.0.1=pyhd8ed1ab_0
- jpeg=9e=hac89ed1_2
- lcms2=2.12=h577c468_0
- lerc=3.0=he49afe7_0
- libblas=3.9.0=15_osx64_openblas
- libcblas=3.9.0=15_osx64_openblas
- libcxx=14.0.6=hce7ea42_0
- libdeflate=1.12=hac89ed1_0
- libffi=3.4.2=h0d85af4_5
- libgfortran=5.0.0=9_3_0_h6c81a4c_23
- libgfortran5=9.3.0=h6c81a4c_23
- liblapack=3.9.0=15_osx64_openblas
- libopenblas=0.3.20=openmp_hb3cd9ec_0
- libpng=1.6.37=h7cec526_2
- libtiff=4.4.0=h9847915_1
- libwebp=1.2.2=h28dabe5_0
- libwebp-base=1.2.2=h0d85af4_1
- libxcb=1.13=h0d85af4_1004
- libzlib=1.2.12=hfe4f2af_1
- llvm-openmp=14.0.4=ha654fa7_0
- locket=1.0.0=pyhd8ed1ab_0
- lz4=4.0.0=py310h2e8d50f_2
- lz4-c=1.9.3=he49afe7_1
- markupsafe=2.1.1=py310h1961e1f_1
- msgpack-python=1.0.4=py310habb735a_0
- multidict=6.0.2=py310h1961e1f_1
- ncurses=6.3=h96cf925_1
- numpy=1.23.0=py310ha3f357c_0
- openjpeg=2.4.0=h6e7aa92_1
- openssl=1.1.1p=hfe4f2af_0
- packaging=21.3=pyhd8ed1ab_0
- pandas=1.4.3=py310h3099161_0
- partd=1.2.0=pyhd8ed1ab_0
- pillow=9.1.1=py310hb3240ae_1
- pip=22.1.2=pyhd8ed1ab_0
- psutil=5.9.1=py310h6c45266_0
- pthread-stubs=0.4=hc929b4f_1001
- pycparser=2.21=pyhd8ed1ab_0
- pyopenssl=22.0.0=pyhd8ed1ab_0
- pyparsing=3.0.9=pyhd8ed1ab_0
- pysocks=1.7.1=py310h2ec42d9_5
- python=3.10.5=hdaaf3db_0_cpython
- python-dateutil=2.8.2=pyhd8ed1ab_0
- python_abi=3.10=2_cp310
- pytz=2022.1=pyhd8ed1ab_0
- pyyaml=6.0=py310h1961e1f_4
- readline=8.1.2=h3899abd_0
- setuptools=63.1.0=py310h2ec42d9_0
- six=1.16.0=pyh6c4a22f_0
- sortedcontainers=2.4.0=pyhd8ed1ab_0
- sqlite=3.39.0=hd9f0692_0
- tblib=1.7.0=pyhd8ed1ab_0
- tk=8.6.12=h5dbffcc_0
- toolz=0.11.2=pyhd8ed1ab_0
- tornado=6.2=py310h6c45266_0
- typing-extensions=4.3.0=hd8ed1ab_0
- typing_extensions=4.3.0=pyha770c72_0
- tzdata=2022a=h191b570_0
- urllib3=1.26.9=pyhd8ed1ab_0
- wheel=0.37.1=pyhd8ed1ab_0
- wrapt=1.14.1=py310h6c45266_0
- xorg-libxau=1.0.9=h35c211d_0
- xorg-libxdmcp=1.1.3=h35c211d_0
- xz=5.2.5=haf1e3a3_1
- yaml=0.2.5=h0d85af4_2
- yarl=1.7.2=py310h1961e1f_2
- zict=2.2.0=pyhd8ed1ab_0
- zlib=1.2.12=hfe4f2af_1
- zstd=1.5.2=ha9df2e0_2
```

Py3.8 environment:
```sh
name: test
channels:
- conda-forge
dependencies:
- aiobotocore=2.3.4=pyhd8ed1ab_0
- aiohttp=3.8.1=py38hed1de0f_1
- aioitertools=0.10.0=pyhd8ed1ab_0
- aiosignal=1.2.0=pyhd8ed1ab_0
- async-timeout=4.0.2=pyhd8ed1ab_0
- attrs=21.4.0=pyhd8ed1ab_0
- bokeh=2.4.3=py38h50d1736_0
- botocore=1.24.21=pyhd8ed1ab_1
- brotlipy=0.7.0=py38hed1de0f_1004
- bzip2=1.0.8=h0d85af4_4
- ca-certificates=2022.6.15=h033912b_0
- certifi=2022.6.15=py38h50d1736_0
- cffi=1.15.1=py38h86886aa_0
- charset-normalizer=2.1.0=pyhd8ed1ab_0
- click=8.1.3=py38h50d1736_0
- cloudpickle=2.1.0=pyhd8ed1ab_0
- cryptography=37.0.2=py38h0f8513e_0
- cytoolz=0.11.2=py38hed1de0f_2
- dask=2022.6.1=pyhd8ed1ab_0
- dask-cloudprovider=2022.5.0=pyhd8ed1ab_0
- dask-core=2022.6.1=pyhd8ed1ab_0
- distributed=2022.6.1=pyhd8ed1ab_0
- freetype=2.10.4=h4cff582_1
- frozenlist=1.3.0=py38hed1de0f_1
- fsspec=2022.5.0=pyhd8ed1ab_0
- giflib=5.2.1=hbcb3906_2
- heapdict=1.0.1=py_0
- idna=3.3=pyhd8ed1ab_0
- jinja2=3.1.2=pyhd8ed1ab_1
- jmespath=1.0.1=pyhd8ed1ab_0
- jpeg=9e=hac89ed1_2
- lcms2=2.12=h577c468_0
- lerc=3.0=he49afe7_0
- libblas=3.9.0=15_osx64_openblas
- libcblas=3.9.0=15_osx64_openblas
- libcxx=14.0.6=hce7ea42_0
- libdeflate=1.12=hac89ed1_0
- libffi=3.4.2=h0d85af4_5
- libgfortran=5.0.0=9_3_0_h6c81a4c_23
- libgfortran5=9.3.0=h6c81a4c_23
- liblapack=3.9.0=15_osx64_openblas
- libopenblas=0.3.20=openmp_hb3cd9ec_0
- libpng=1.6.37=h7cec526_2
- libtiff=4.4.0=h9847915_1
- libwebp=1.2.2=h28dabe5_0
- libwebp-base=1.2.2=h0d85af4_1
- libxcb=1.13=h0d85af4_1004
- libzlib=1.2.12=hfe4f2af_1
- llvm-openmp=14.0.4=ha654fa7_0
- locket=1.0.0=pyhd8ed1ab_0
- lz4=4.0.0=py38h5cd37e2_2
- lz4-c=1.9.3=he49afe7_1
- markupsafe=2.1.1=py38hed1de0f_1
- msgpack-python=1.0.4=py38hf58141a_0
- multidict=6.0.2=py38hed1de0f_1
- ncurses=6.3=h96cf925_1
- numpy=1.23.0=py38h604f2a5_0
- openjpeg=2.4.0=h6e7aa92_1
- openssl=1.1.1p=hfe4f2af_0
- packaging=21.3=pyhd8ed1ab_0
- pandas=1.4.3=py38h2b30649_0
- partd=1.2.0=pyhd8ed1ab_0
- pillow=9.1.1=py38h21af888_1
- pip=22.1.2=pyhd8ed1ab_0
- psutil=5.9.1=py38h0dd4459_0
- pthread-stubs=0.4=hc929b4f_1001
- pycparser=2.21=pyhd8ed1ab_0
- pyopenssl=22.0.0=pyhd8ed1ab_0
- pyparsing=3.0.9=pyhd8ed1ab_0
- pysocks=1.7.1=py38h50d1736_5
- python=3.8.13=h394c593_0_cpython
- python-dateutil=2.8.2=pyhd8ed1ab_0
- python_abi=3.8=2_cp38
- pytz=2022.1=pyhd8ed1ab_0
- pyyaml=6.0=py38hed1de0f_4
- readline=8.1.2=h3899abd_0
- setuptools=63.1.0=py38h50d1736_0
- six=1.16.0=pyh6c4a22f_0
- sortedcontainers=2.4.0=pyhd8ed1ab_0
- sqlite=3.39.0=hd9f0692_0
- tblib=1.7.0=pyhd8ed1ab_0
- tk=8.6.12=h5dbffcc_0
- toolz=0.11.2=pyhd8ed1ab_0
- tornado=6.2=py38h0dd4459_0
- typing-extensions=4.3.0=hd8ed1ab_0
- typing_extensions=4.3.0=pyha770c72_0
- tzdata=2022a=h191b570_0
- urllib3=1.26.9=pyhd8ed1ab_0
- wheel=0.37.1=pyhd8ed1ab_0
- wrapt=1.14.1=py38h0dd4459_0
- xorg-libxau=1.0.9=h35c211d_0
- xorg-libxdmcp=1.1.3=h35c211d_0
- xz=5.2.5=haf1e3a3_1
- yaml=0.2.5=h0d85af4_2
- yarl=1.7.2=py38hed1de0f_2
- zict=2.2.0=pyhd8ed1ab_0
- zlib=1.2.12=hfe4f2af_1
- zstd=1.5.2=ha9df2e0_2
```
Using the following script:
```py
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import logging
from time import sleep

from dask_cloudprovider.aws import FargateCluster, ECSCluster
from distributed import Client

###############################################################################

logging.basicConfig(
level=logging.INFO,
format="[%(levelname)4s: %(module)s:%(lineno)4s %(asctime)s] %(message)s",
)
log = logging.getLogger(__name__)

###############################################################################

def add_one(val: int) -> int:
sleep(0.1)
return val + 1

def multiply_by_ten(val: int) -> int:
sleep(0.1)
return val * 10

def main():
# Cluster setup
# https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html#fargate-tasks-size
cluster = FargateCluster(fargate_spot=True, worker_mem=512, worker_cpu=256, n_workers=3)

log.info(f"Cluster name is {cluster.name}")
log.info(f"Dashboard available at: {cluster.dashboard_link}")
client = Client(cluster.scheduler_address)

# Actual work
initial_values = list(range(100))
log.info(f"Values range: [{initial_values[0]}-{initial_values[-1]}]")

# Add
addition_futures = client.map(add_one, initial_values)
values = client.gather(addition_futures)

# Multiply
multiplication_futures = client.map(multiply_by_ten, values)
values = client.gather(multiplication_futures)

# Sort
values = sorted(values)

# Results
log.info(f"Values range: [{values[0]}-{values[-1]}]")

log.info("All done!")
client.close()
cluster.close()

if __name__ == "__main__":
main()
```

Full Stacktrace:
```
Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/client.py:1274: VersionMismatchWarning: Mismatched versions found

+---------+----------------+----------------+----------------+
| Package | client | scheduler | workers |
+---------+----------------+----------------+----------------+
| python | 3.10.5.final.0 | 3.8.13.final.0 | 3.8.13.final.0 |
| tornado | 6.2 | 6.1 | 6.1 |
+---------+----------------+----------------+----------------+
warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
[INFO: client: 45 2022-07-04 16:30:16,057] Values range: [0-99]
[INFO: client: 59 2022-07-04 16:30:23,609] Values range: [10-1000]
[INFO: client: 61 2022-07-04 16:30:23,609] All done!
Traceback (most recent call last):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 178, in send
response = await self._session.request(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 542, in connect
proto = await self._create_connection(req, traces, timeout)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 907, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 1154, in _create_direct_connection
hosts = await asyncio.shield(host_resolved)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 880, in _resolve_host
addrs = await self._resolver.resolve(host, port, family=self._family)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
infos = await self._loop.getaddrinfo(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 860, in getaddrinfo
return await self.run_in_executor(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 818, in run_in_executor
executor.submit(func, *args), loop=self)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 667, in _exitfunc
f()
File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 591, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 336, in sync
return sync(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 403, in sync
raise exc.with_traceback(tb)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 376, in f
result = yield future
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
value = future.result()
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/dask_cloudprovider/aws/ecs.py", line 1313, in _delete_worker_task_definition_arn
await ecs.deregister_task_definition(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 251, in _make_api_call
http, parsed_response = await self._make_request(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 271, in _make_request
return await self._endpoint.make_request(operation_model, request_dict)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 80, in _send_request
while await self._needs_retry(attempts, operation_model,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 213, in _needs_retry
responses = await self._event_emitter.emit(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/hooks.py", line 50, in _emit
response = await resolve_awaitable(handler(**kwargs))
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 89, in _call
if await resolve_awaitable(self._checker(**checker_kwargs)):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 107, in _call
should_retry = await self._should_retry(attempt_number, response,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 129, in _should_retry
return await resolve_awaitable(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 146, in _call
checker(attempt_number, response,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 231, in __call__
return self._check_caught_exception(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 374, in _check_caught_exception
raise caught_exception
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 149, in _do_get_response
http_response = await self._send(request)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 229, in _send
return await self.http_session.send(request)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 220, in send
raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: cannot schedule new futures after shutdown
Traceback (most recent call last):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 178, in send
response = await self._session.request(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 542, in connect
proto = await self._create_connection(req, traces, timeout)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 907, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 1154, in _create_direct_connection
hosts = await asyncio.shield(host_resolved)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 880, in _resolve_host
addrs = await self._resolver.resolve(host, port, family=self._family)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
infos = await self._loop.getaddrinfo(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 860, in getaddrinfo
return await self.run_in_executor(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 818, in run_in_executor
executor.submit(func, *args), loop=self)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 667, in _exitfunc
f()
File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 591, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 336, in sync
return sync(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 403, in sync
raise exc.with_traceback(tb)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 376, in f
result = yield future
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
value = future.result()
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/dask_cloudprovider/aws/ecs.py", line 1240, in _delete_scheduler_task_definition_arn
await ecs.deregister_task_definition(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 251, in _make_api_call
http, parsed_response = await self._make_request(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 271, in _make_request
return await self._endpoint.make_request(operation_model, request_dict)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 80, in _send_request
while await self._needs_retry(attempts, operation_model,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 213, in _needs_retry
responses = await self._event_emitter.emit(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/hooks.py", line 50, in _emit
response = await resolve_awaitable(handler(**kwargs))
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 89, in _call
if await resolve_awaitable(self._checker(**checker_kwargs)):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 107, in _call
should_retry = await self._should_retry(attempt_number, response,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 129, in _should_retry
return await resolve_awaitable(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 146, in _call
checker(attempt_number, response,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 231, in __call__
return self._check_caught_exception(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 374, in _check_caught_exception
raise caught_exception
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 149, in _do_get_response
http_response = await self._send(request)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 229, in _send
return await self.http_session.send(request)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 220, in send
raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: cannot schedule new futures after shutdown
Traceback (most recent call last):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 178, in send
response = await self._session.request(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 542, in connect
proto = await self._create_connection(req, traces, timeout)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 907, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 1154, in _create_direct_connection
hosts = await asyncio.shield(host_resolved)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 880, in _resolve_host
addrs = await self._resolver.resolve(host, port, family=self._family)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
infos = await self._loop.getaddrinfo(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 860, in getaddrinfo
return await self.run_in_executor(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 818, in run_in_executor
executor.submit(func, *args), loop=self)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 667, in _exitfunc
f()
File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 591, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 336, in sync
return sync(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 403, in sync
raise exc.with_traceback(tb)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 376, in f
result = yield future
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
value = future.result()
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/dask_cloudprovider/aws/ecs.py", line 1130, in _delete_role
attached_policies = (await iam.list_attached_role_policies(RoleName=role))[
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 251, in _make_api_call
http, parsed_response = await self._make_request(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 271, in _make_request
return await self._endpoint.make_request(operation_model, request_dict)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 80, in _send_request
while await self._needs_retry(attempts, operation_model,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 213, in _needs_retry
responses = await self._event_emitter.emit(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/hooks.py", line 50, in _emit
response = await resolve_awaitable(handler(**kwargs))
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 89, in _call
if await resolve_awaitable(self._checker(**checker_kwargs)):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 107, in _call
should_retry = await self._should_retry(attempt_number, response,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 129, in _should_retry
return await resolve_awaitable(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 146, in _call
checker(attempt_number, response,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 231, in __call__
return self._check_caught_exception(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 374, in _check_caught_exception
raise caught_exception
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 149, in _do_get_response
http_response = await self._send(request)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 229, in _send
return await self.http_session.send(request)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 220, in send
raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: cannot schedule new futures after shutdown
Traceback (most recent call last):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 178, in send
response = await self._session.request(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 542, in connect
proto = await self._create_connection(req, traces, timeout)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 907, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 1154, in _create_direct_connection
hosts = await asyncio.shield(host_resolved)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 880, in _resolve_host
addrs = await self._resolver.resolve(host, port, family=self._family)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
infos = await self._loop.getaddrinfo(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 860, in getaddrinfo
return await self.run_in_executor(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 818, in run_in_executor
executor.submit(func, *args), loop=self)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 667, in _exitfunc
f()
File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 591, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 336, in sync
return sync(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 403, in sync
raise exc.with_traceback(tb)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 376, in f
result = yield future
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
value = future.result()
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/dask_cloudprovider/aws/ecs.py", line 1130, in _delete_role
attached_policies = (await iam.list_attached_role_policies(RoleName=role))[
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 251, in _make_api_call
http, parsed_response = await self._make_request(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 271, in _make_request
return await self._endpoint.make_request(operation_model, request_dict)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 80, in _send_request
while await self._needs_retry(attempts, operation_model,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 213, in _needs_retry
responses = await self._event_emitter.emit(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/hooks.py", line 50, in _emit
response = await resolve_awaitable(handler(**kwargs))
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 89, in _call
if await resolve_awaitable(self._checker(**checker_kwargs)):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 107, in _call
should_retry = await self._should_retry(attempt_number, response,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 129, in _should_retry
return await resolve_awaitable(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 146, in _call
checker(attempt_number, response,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 231, in __call__
return self._check_caught_exception(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 374, in _check_caught_exception
raise caught_exception
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 149, in _do_get_response
http_response = await self._send(request)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 229, in _send
return await self.http_session.send(request)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 220, in send
raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: cannot schedule new futures after shutdown
Traceback (most recent call last):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 178, in send
response = await self._session.request(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 542, in connect
proto = await self._create_connection(req, traces, timeout)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 907, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 1154, in _create_direct_connection
hosts = await asyncio.shield(host_resolved)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 880, in _resolve_host
addrs = await self._resolver.resolve(host, port, family=self._family)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
infos = await self._loop.getaddrinfo(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 860, in getaddrinfo
return await self.run_in_executor(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 818, in run_in_executor
executor.submit(func, *args), loop=self)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 667, in _exitfunc
f()
File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 591, in __call__
return info.func(*info.args, **(info.kwargs or {}))
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 336, in sync
return sync(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 403, in sync
raise exc.with_traceback(tb)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 376, in f
result = yield future
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
value = future.result()
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/dask_cloudprovider/aws/ecs.py", line 1049, in _delete_cluster
async for page in ecs.get_paginator("list_tasks").paginate(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/paginate.py", line 32, in __anext__
response = await self._make_request(current_kwargs)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 251, in _make_api_call
http, parsed_response = await self._make_request(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 271, in _make_request
return await self._endpoint.make_request(operation_model, request_dict)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 80, in _send_request
while await self._needs_retry(attempts, operation_model,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 213, in _needs_retry
responses = await self._event_emitter.emit(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/hooks.py", line 50, in _emit
response = await resolve_awaitable(handler(**kwargs))
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 89, in _call
if await resolve_awaitable(self._checker(**checker_kwargs)):
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 107, in _call
should_retry = await self._should_retry(attempt_number, response,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 129, in _should_retry
return await resolve_awaitable(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
return await obj
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 146, in _call
checker(attempt_number, response,
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 231, in __call__
return self._check_caught_exception(
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 374, in _check_caught_exception
raise caught_exception
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 149, in _do_get_response
http_response = await self._send(request)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 229, in _send
return await self.http_session.send(request)
File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 220, in send
raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: cannot schedule new futures after shutdown
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.