dask / dask/dask-cloudprovider

Azure: AttributeError when attempting authentication

Open
#329 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

**What happened**: Followed the instructions to try out Dask with Azure from [the documentation here](https://cloudprovider.dask.org/en/latest/azure.html). Installed `azure-cli` and `dask-cloudprovider[azure]` and tried the minimal example that resulted in a traceback (for reference `create.py` holds the minimal reproducible code also pasted in this ticket)

```
python create.py
/Users/alfredo/code/dask-demo/create.py:2: FutureWarning: serialize_for_cli is deprecated and will be removed in a future release. Please use dask.config.serialize instead.
from distributed.utils import warn_on_duration, serialize_for_cli, cli_keywords
Creating scheduler instance
Traceback (most recent call last):
File "/Users/alfredo/code/dask-demo/create.py", line 6, in
cluster = AzureVMCluster(location="eastus", resource_group="ml-demo", vnet="dask-vnet", security_group="dask-nsg", n_workers=4)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/dask_cloudprovider/azure/azurevm.py", line 550, in __init__
super().__init__(debug=debug, **kwargs)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/dask_cloudprovider/generic/vmcluster.py", line 289, in __init__
super().__init__(**kwargs, security=self.security)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/distributed/deploy/spec.py", line 260, in __init__
self.sync(self._start)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/distributed/utils.py", line 310, in sync
return sync(
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/distributed/utils.py", line 364, in sync
raise exc.with_traceback(tb)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/distributed/utils.py", line 349, in f
result[0] = yield future
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/dask_cloudprovider/generic/vmcluster.py", line 329, in _start
await super()._start()
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/distributed/deploy/spec.py", line 289, in _start
self.scheduler = await self.scheduler
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/distributed/deploy/spec.py", line 59, in _
await self.start()
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/dask_cloudprovider/generic/vmcluster.py", line 86, in start
ip = await self.create_vm()
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/dask_cloudprovider/azure/azurevm.py", line 72, in create_vm
[subnet_info, *_] = await self.cluster.call_async(
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/core/paging.py", line 129, in __next__
return next(self._page_iterator)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/core/paging.py", line 76, in __next__
self._response = self._get_next(self.continuation_token)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/mgmt/network/v2021_05_01/operations/_subnets_operations.py", line 689, in get_next
pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 211, in run
return first_node.send(pipeline_request) # type: ignore
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 71, in send
response = self.next.send(request)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/mgmt/core/policies/_base.py", line 47, in send
response = self.next.send(request)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 71, in send
response = self.next.send(request)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 71, in send
response = self.next.send(request)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/core/pipeline/_base.py", line 71, in send
response = self.next.send(request)
[Previous line repeated 1 more time]
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/core/pipeline/policies/_redirect.py", line 158, in send
response = self.next.send(request)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/core/pipeline/policies/_retry.py", line 445, in send
response = self.next.send(request)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/core/pipeline/policies/_authentication.py", line 117, in send
self.on_request(request)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/core/pipeline/policies/_authentication.py", line 94, in on_request
self._token = self._credential.get_token(*self._scopes)
File "/Users/alfredo/miniforge3/envs/dask/lib/python3.10/site-packages/azure/common/credentials.py", line 72, in get_token
_, token, fulltoken = credentials._token_retriever() # pylint:disable=protected-access
AttributeError: 'CredentialAdaptor' object has no attribute '_token_retriever'
```

**What you expected to happen**: Full compatibility with the credential object

**Minimal Complete Verifiable Example**:

Install `dask-cloudprovider[azure]`, `azure-cli`, run `azure login` and then try the following with existing resource groups, vnet, and security group:

```python
from dask_cloudprovider.azure import AzurePreemptibleWorkerPlugin
from distributed.utils import warn_on_duration, serialize_for_cli, cli_keywords
from distributed import Client
from dask_cloudprovider.azure import AzureVMCluster

cluster = AzureVMCluster(location="eastus", resource_group="***", vnet="****", security_group="***", n_workers=4)
```
**Anything else we need to know?**: This seems to have been reported in the [Azure repo as an issue](https://github.com/Azure/azure-sdk-for-python/issues/22073). Specifically, there is an incompatibility [described in this comment](https://github.com/Azure/azure-sdk-for-python/issues/22073#issuecomment-991311505).

I was unable to install the referred to versions as I started getting into unresolvable conflicts.

It would be ideal to have pinned dependencies, or restrictive versioning in `setup.py` instead of the current scheme which allows for anything new to be used, which has greater potential for disruption:

```python
"azure": [
"azure-mgmt-compute>=18.0.0",
"azure-mgmt-network>=16.0.0",
"azure-cli-core>=2.15.1",
"msrestazure",
"azure-identity",
],
```

**Environment**:

- Dask version: dask==2021.12.0
- Python version: Python 3.10.1
- Operating System: OSX Apple silicon, Monterer 12.0.1
- Install method (conda, pip, source): It is a conda environment but installed everything with `pip`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.