dask / dask/dask-cloudprovider

AzureVMCluster workers unable to connect to scheduler

Open
#254 7 comments 0 reactions 0 assignees View on GitHub
bug provider/azure/vm
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

**What happened**:
I'm just getting started with dask-cloudprovider, and looking to spin up an `AzureVMCluster` to parallelise my dask workloads. I followed the [documentation](https://cloudprovider.dask.org/en/latest/azure.html), to create the required infrastructure (resource group, network security group, vnet) except that, rather than allowing network traffic from the internet to ports 8786-8787, I restricted the IP to be the one of my local machine.

Following the minimal example in the documentation, i.e:
```python
from dask_cloudprovider.azure import AzureVMCluster

cluster = AzureVMCluster(
location=LOCATION,
resource_group=RESOURCE_GROUP,
vnet=VNET,
security_group=SECURITY_GROUP,
n_workers=1
)
```
I find that I am able to successfully create the cluster, and am able to access the web dashboard (confirming that my network security group rule is working correctly). However, I find that my workers do not seem to be able to connect to the cluster. The only way I can appear to get them to connect is by creating an additional rule which allows access via the internet, as in the original example (obviously undesirable).

In the network security group, there are the default rules in place which allow incoming traffic across the vnet (which should cover the worker/scheduler connection). Adding a specific rule which allows traffic from the private ip range (10.0.0.0/24) doesn't help either.

(Strangely, if I create the "allow internet access" rule to allow the workers to be discovered by the scheduler, if I then remove the rule, the computation is still able to proceed, so I suspect there might be something strange happening with the worker discoverability?)

**What you expected to happen**:
The workers should be able to connect to the scheduler without allowing unrestricted access to the vnet.

**Environment**:

- Dask cloud provider version: 2021.01.1
- Python version: 3.7
- Operating System: macOS 11
- Install method (conda, pip, source): pip

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.