dask / dask/dask-cloudprovider

VMClusters can hang if something fails during scheduler provisioning

Open
#277 10 comments 0 reactions 0 assignees View on GitHub
bug provider/aws/ec2 provider/azure/vm provider/digitalocean/droplet provider/gcp/vm
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

When launching clusters that extend `VMCluster` there is a failure mode where the cluster manager can just hang on creation, perhaps indefinitely. See #275.

The reason this happens is that we submit the Dask scheduler to each cloud provider as a VM with a preconfigured startup script using [cloud init](https://cloudinit.readthedocs.io/en/latest/). You can view this script by setting the `debug=True` flag as a kwarg on the cluster manager, the init script will be printed when the VM is submitted.

One benefit of launching our work via cloud init is that we don't actually need to establish any inbound connection such as SSH to the VM in order to provision it. This helps keep the attack surface small and means we do not need to manage secrets in Dask Cloudprovider such as SSH keys.

The VM runs the cloud init script on startup and runs Dask. We wait for this to come up and then connect to the Dask comm on port `8786` (or whatever port we assign) and continue working from there. One drawback to this is that we submit our VM in a fire-and-forget way and then have to wait until the Dask comm port becomes available. If Dask fails to start we will end up waiting forever.

This isn't a pleasant experience for users as it is unclear whether the VM is just taking a long time to start or has failed. VMs can take some time to start because of capacity on the cloud provider or perhaps the user selected a large Docker image that takes a long time to download and decompress.

Things we should do to improve this today:
- Add a timeout with automatic cleanup of the VM. We should avoid leaving hanging resources in case of failure and raise an exception to the user that the scheduler timed out starting up.

Design change options we could make to improve the debugging workflow:

1. Instead of provisioning via cloud init we configure SSH and configure Dask over SSH, allowing failure information to be fed back to the user more easily.
1. Keep provisioning with cloud init but make more of an effort to enable SSH to allow for interactive debugging.
1. Run some log exporting service on each VM to pass the contents of `/var/log/cloud-init-output.log` back to the cluster manager and on to the user. Typically to debug the hanging problem today we just need to see what is in this log file, so passing it back as a default operation could help.

I have concerns about trying to use SSH. Many orgs will have policies around SSH in terms of how the port is exposed, how keys are managed, whether you must use a bastion, etc. So while I think it is a good idea to enable SSH where possible I'm hesitant to depend on it for provisioning.

Exporting logs may have some drawbacks too. The simplest implementation of this would be a web server which serves the log files we want to read, but we would need to expose an additional port for this and ensure it is protected in some way.

My current leaning is to try and expose SSH as an optional feature but go with option 3 to resolve this issue. But feedback and discussion would be welcome.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.