dask / dask/dask-cloudprovider
Spot instance support for EC2
- Dominant language
- Python
- Stars
- 147
- Forks
- 119
- PR merge metrics
- No merged PRs in 30d
Description
This works from a touch test, will keep working with it here, but interested in any feedback / collaboration.
Seems critical to support, as this should be a 4-5x reduction in cost.
Once I finally figured out what to change, was fairly simple, concerns are:
1. We don't want a spot instance for the scheduler. checking if worker_class is present seems to work, but is not elegant.
2. Unclear what behaviour will be when spot instances are reclaimed. I'm hoping it will gracefully recover and delete the worker?
For reference: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_SpotMarketOptions.html
```diff
diff --git a/dask_cloudprovider/aws/ec2.py b/dask_cloudprovider/aws/ec2.py
index 1a4f27d..486eeb6 100644
--- a/dask_cloudprovider/aws/ec2.py
+++ b/dask_cloudprovider/aws/ec2.py
@@ -85,7 +85,6 @@ class EC2Instance(VMInterface):
"""
https://botocore.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.run_instances
"""
- # TODO Enable Spot support
+ spot = hasattr(self, "worker_class")
@@ -152,6 +151,15 @@ class EC2Instance(VMInterface):
],
}
+ if spot:
+ vm_kwargs['InstanceMarketOptions'] = {
+ 'MarketType': 'spot',
+ 'SpotOptions': {
+ 'SpotInstanceType': 'one-time',
+ }
+ }
+
if self.key_name:
vm_kwargs["KeyName"] = self.key_name
```
Contributor guide
Assessment
This issue has not been assessed yet.