dask / dask/dask-cloudprovider
`EC2Cluster` support for multiple instance types
- Dominant language
- Python
- Stars
- 147
- Forks
- 119
- PR merge metrics
- No merged PRs in 30d
Description
## Background
With large clusters of popular instance types, it's not uncommon to get `InsufficientInstanceCapacity` errors from AWS. This request is for a feature to help avoid/circumvent these errors.
## Desired Feature
Instead of specifying one worker instance type, one could instead specify a (sorted) list of instance types [X, Y, Z, ...]. If 100 workers are requested, we attempt to create 100 instances of type X. If N1 instances are created then we receive a capacity error, we request (100-N1) instances of type Y. This continues until all instance types are exhausted. Of course, we could still be left with fewer than the desired number of workers, but this approach increases our chances of success.
Note: the burden is on the user to ensure that the instance types are appropriate for the use case. For example, users would probably want to ensure that instances types 2, 3, ... have at least as many vCPU and memory as type 1. This is definitely a "super user" type of feature.
## Proposed Implementation
To maintain backwards compatibility, we can update `worker_instance_type` to accept either a string or a list of strings. If we are not worried about backwards compatibility, we can change the argument to `worker_instance_types` and require a list, but this seems like an unnecessary breaking change.
I'm open to other implementations if you think the idea is worth working on. I'm also happy to contribute a solution here.
Contributor guide
Assessment
This issue has not been assessed yet.