dask / dask/dask-cloudprovider

`EC2Cluster` support for multiple instance types

Open
#380 1 comment 0 reactions 0 assignees View on GitHub
enhancement help wanted provider/aws/ecs
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

## Background

With large clusters of popular instance types, it's not uncommon to get `InsufficientInstanceCapacity` errors from AWS. This request is for a feature to help avoid/circumvent these errors.

## Desired Feature

Instead of specifying one worker instance type, one could instead specify a (sorted) list of instance types [X, Y, Z, ...]. If 100 workers are requested, we attempt to create 100 instances of type X. If N1 instances are created then we receive a capacity error, we request (100-N1) instances of type Y. This continues until all instance types are exhausted. Of course, we could still be left with fewer than the desired number of workers, but this approach increases our chances of success.

Note: the burden is on the user to ensure that the instance types are appropriate for the use case. For example, users would probably want to ensure that instances types 2, 3, ... have at least as many vCPU and memory as type 1. This is definitely a "super user" type of feature.

## Proposed Implementation

To maintain backwards compatibility, we can update `worker_instance_type` to accept either a string or a list of strings. If we are not worried about backwards compatibility, we can change the argument to `worker_instance_types` and require a list, but this seems like an unnecessary breaking change.

I'm open to other implementations if you think the idea is worth working on. I'm also happy to contribute a solution here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.