dask / dask/dask-cloudprovider

Add the ability to configure EC2Cluster to start a publicly accessible Scheduler while keeping Workers private

Open
#388 1 comment 0 reactions 0 assignees View on GitHub
enhancement help wanted provider/aws/ec2
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

For existing discussion, see [this](https://dask.discourse.group/t/how-to-pass-private-ip-of-scheduler-to-dask-worker-running-on-ec2/1259/4) Discourse thread.

---
I have a use case where I wish to develop something locally and then start up a `dask_cloudprovider.aws.EC2Cluster` such that all Scheduler/Worker communication happens on the internal EC2 Network (i.e. via the private VPC/Subnet IP addresses), but still allow me to set up a rule so that my IP can access the scheduler Dashboard.

The current normal development flow is to deploy this cluster from within AWS. This is a bit burdensome since it means I need to manage a separate VM and also code deployment to this VM (or rebuilding an image every time I make a code change).

Currently, when I start an `EC2Cluster` with `use_private_ip=False`, the Scheduler advertises its *public* IP to the workers. A relatively straightforward security group configuration (specific to AWS) might be:

| Port range | Source |
| -------- | ------------ |
| 8786 - 8787 | [own group] |
| 8786 - 8787 | [my ip] |

which would technically allow the communication described above, but does not work since it does not allow public IP<->public IP communication. Creating this group is probably outside of the scope of responsibilities for `dask_cloudprovider` (as opposed to the `dask-default` one), and it would suffice to be able to use it as given above.

According to @jacobtomlinson , it might already be possible to do this with an [ECSCluster](https://cloudprovider.dask.org/en/stable/aws.html#elastic-container-service-ecs), though I haven't verified that functionality.

---
As a side note, I was able to get this working locally by monkey-patching `dask_cloudprovider.aws.ec2#configure_vm` to return `instance['PublicDnsName']` instead of `instance['PublicIpAddress']`, but this is a super hacky workaround that:
- Only works because AWS resolves DNS as the public IP externally and private IP internally
- Requires the workers to have public IP addresses that it does not need (and are not reachable)

The solution to this would likely not use anything like the above, but I thought the information might be helpful in this context.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.