dask / dask/dask-cloudprovider
Error on RegisterTaskDefinition for worker when instantiating a FargateCluster
- Dominant language
- Python
- Stars
- 147
- Forks
- 119
- PR merge metrics
- No merged PRs in 30d
Description
Occasionally I see the following error when dask cloudprovider attempts to create a worker task definition while creating a Fargate cluster:
```
Unexpected error: ClientException('An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.')
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/prefect/engine/runner.py", line 48, in inner
new_state = method(self, state, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/prefect/engine/flow_runner.py", line 442, in get_flow_run_state
with self.check_for_cancellation(), executor.start():
File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.8/dist-packages/prefect/executors/dask.py", line 223, in start
with self.cluster_class(**self.cluster_kwargs) as cluster:
File "/usr/local/lib/python3.8/dist-packages/dask_cloudprovider/aws/ecs.py", line 1367, in __init__
super().__init__(fargate_scheduler=True, fargate_workers=True, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/dask_cloudprovider/aws/ecs.py", line 733, in __init__
super().__init__(**kwargs)
File "/usr/local/lib/python3.8/dist-packages/distributed/deploy/spec.py", line 282, in __init__
self.sync(self._start)
File "/usr/local/lib/python3.8/dist-packages/distributed/deploy/cluster.py", line 188, in sync
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/distributed/utils.py", line 353, in sync
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.8/dist-packages/distributed/utils.py", line 336, in f
result[0] = yield future
File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/local/lib/python3.8/dist-packages/dask_cloudprovider/aws/ecs.py", line 887, in _start
await self._create_worker_task_definition_arn()
File "/usr/local/lib/python3.8/dist-packages/dask_cloudprovider/aws/ecs.py", line 1159, in _create_worker_task_definition_arn
response = await ecs.register_task_definition(
File "/usr/local/lib/python3.8/dist-packages/aiobotocore/client.py", line 154, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.
```
Contributor guide
Assessment
This issue has not been assessed yet.