dask / dask/dask-cloudprovider

Spot instance support for EC2

Open
#366 1 comment 2 reactions 0 assignees View on GitHub
enhancement help wanted provider/aws/ec2
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

This works from a touch test, will keep working with it here, but interested in any feedback / collaboration.
Seems critical to support, as this should be a 4-5x reduction in cost.
Once I finally figured out what to change, was fairly simple, concerns are:

1. We don't want a spot instance for the scheduler. checking if worker_class is present seems to work, but is not elegant.
2. Unclear what behaviour will be when spot instances are reclaimed. I'm hoping it will gracefully recover and delete the worker?

For reference: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_SpotMarketOptions.html

```diff
diff --git a/dask_cloudprovider/aws/ec2.py b/dask_cloudprovider/aws/ec2.py
index 1a4f27d..486eeb6 100644
--- a/dask_cloudprovider/aws/ec2.py
+++ b/dask_cloudprovider/aws/ec2.py
@@ -85,7 +85,6 @@ class EC2Instance(VMInterface):
"""
https://botocore.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.run_instances
"""
- # TODO Enable Spot support

+ spot = hasattr(self, "worker_class")
@@ -152,6 +151,15 @@ class EC2Instance(VMInterface):
],
}

+ if spot:
+ vm_kwargs['InstanceMarketOptions'] = {
+ 'MarketType': 'spot',
+ 'SpotOptions': {
+ 'SpotInstanceType': 'one-time',
+ }
+ }
+
if self.key_name:
vm_kwargs["KeyName"] = self.key_name

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.