dask / dask/dask-cloudprovider

Proposal: New implementation of ECSCluster

Open
#330 3 comments 5 reactions 0 assignees View on GitHub
provider/aws/ecs
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

I use `ECSCluster` heavily, but I find it has lots of small implementation issues that make it hard to reliably run large Dask clusters. Here's a laundry list of some of the problems I've run into.

* Task names are always derived from the cluster name. In a shared ECS cluster, task definitions overlap between multiple ECSCluster instances, making it impossible to tell what tasks belong to which Dask cluster.
* API rate limits are not handled properly. Combined with the log parsing for addresses (relates to #313, #121), large clusters are hard to reliably instantiate because the worker IP addresses can't be found.
* ECSCluster directly instantiates tasks without using a service. There's no good way to do placement strategies like binpack.
* Exited tasks are not handled and rescheduled. When workers run on spot instances, the Dask cluster can gradually lose workers as spot instances come and go.
* There's no way to configure capacity providers for tasks.
* There's no way to configure different subnets, environment variables, etc. for schedulers vs workers.
* There's no way to configure driver to something other than awslogs.
* Scaling a cluster while a previous scale is still in progress sometimes fails.
* Too many IAM permissions are required, even when using pre-existing ECS clusters and resources.
* Deprovisioning of tasks for both workers and schedulers is not clean (relates to #262).
* Closing the client and cluster objects results in dangling hooks.

Rather than trying to morph the existing `ECSCluster` class, would this project be open to a completely new implementation (`ECSCluster2`?). I anticipate API changes are required (i.e., the arguments to `ECSCluster`). I'm willing to tackle this myself.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.