dask / dask/dask-cloudprovider

DefaultAzureCredential failed to retrieve a token from the included credentials.

Open
#340 0 comments 2 reactions 0 assignees View on GitHub
bug provider/azure/vm
Dominant language
Python
Stars
147
Forks
119
PR merge metrics
No merged PRs in 30d

Description

**What happened**:
After following the [authentication instructions](https://cloudprovider.dask.org/en/latest/azure.html#overview) using the Azure CLI, I'm unable to create a cluster on Azure.

**Minimal Complete Verifiable Example**:

```python
cluster = AzureVMCluster(
location=,
resource_group=,
vnet=,
security_group=,
n_workers=1)
```

**Anything else we need to know?**:

```shell
DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot.this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
SharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
VisualStudioCodeCredential: Failed to get Azure user details from Visual Studio Code.
AzureCliCredential: Unexpected output from Azure CLI: 'bash: /opt/conda/envs/rapids/lib/libtinfo.so.6: no version information available (required by bash)
{
****,
"expiresOn": "2022-03-28 18:00:36.000000",
"subscription": "#####",
"tenant": "#####",
"tokenType": "Bearer"
}
'.
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/azclicredential/troubleshoot.
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
Exception ignored in:
Traceback (most recent call last):
File "/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/deploy/cluster.py", line 196, in __del__
if self.status != Status.closed:
AttributeError: 'AzureVMCluster' object has no attribute 'status'
---------------------------------------------------------------------------
ClientAuthenticationError Traceback (most recent call last)
in

/opt/conda/envs/rapids/lib/python3.8/site-packages/dask_cloudprovider/azure/azurevm.py in __init__(self, location, resource_group, vnet, security_group, public_ingress, vm_size, scheduler_vm_size, vm_image, disk_size, bootstrap, auto_shutdown, docker_image, debug, marketplace_plan, **kwargs)
554 }
555 self.worker_options = {"vm_size": self.vm_size, **self.options}
--> 556 super().__init__(debug=debug, **kwargs)

/opt/conda/envs/rapids/lib/python3.8/site-packages/dask_cloudprovider/generic/vmcluster.py in __init__(self, n_workers, worker_class, worker_options, scheduler_options, docker_image, docker_args, extra_bootstrap, env_vars, security, protocol, debug, **kwargs)
294 self.uuid = str(uuid.uuid4())[:8]
295
--> 296 super().__init__(**kwargs, security=self.security)
297
298 async def call_async(self, f, *args, **kwargs):

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/deploy/spec.py in __init__(self, workers, scheduler, worker, asynchronous, loop, security, silence_logs, name, shutdown_on_close, scheduler_sync_interval)
258 if not self.asynchronous:
259 self._loop_runner.start()
--> 260 self.sync(self._start)
261 try:
262 self.sync(self._correct_state)

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/utils.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
307 return future
308 else:
--> 309 return sync(
310 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
311 )

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
374 if error:
375 typ, exc, tb = error
--> 376 raise exc.with_traceback(tb)
377 else:
378 return result

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/utils.py in f()
347 future = asyncio.wait_for(future, callback_timeout)
348 future = asyncio.ensure_future(future)
--> 349 result = yield future
350 except Exception:
351 error = sys.exc_info()

/opt/conda/envs/rapids/lib/python3.8/site-packages/tornado/gen.py in run(self)
760
761 try:
--> 762 value = future.result()
763 except Exception:
764 exc_info = sys.exc_info()

/opt/conda/envs/rapids/lib/python3.8/site-packages/dask_cloudprovider/generic/vmcluster.py in _start(self)
336 "Hang tight! ",
337 ):
--> 338 await super()._start()
339
340 def render_process_cloud_init(self, process):

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/deploy/spec.py in _start(self)
291 cls = import_term(cls)
292 self.scheduler = cls(**self.scheduler_spec.get("options", {}))
--> 293 self.scheduler = await self.scheduler
294 self.scheduler_comm = rpc(
295 getattr(self.scheduler, "external_address", None) or self.scheduler.address,

/opt/conda/envs/rapids/lib/python3.8/site-packages/distributed/deploy/spec.py in _()
57 async with self.lock:
58 if self.status == Status.created:
---> 59 await self.start()
60 assert self.status == Status.running
61 return self

/opt/conda/envs/rapids/lib/python3.8/site-packages/dask_cloudprovider/generic/vmcluster.py in start(self)
85 async def start(self):
86 self.cluster._log("Creating scheduler instance")
---> 87 ip = await self.create_vm()
88 self.address = f"{self.cluster.protocol}://{ip}:8786"
89 await self.wait_for_scheduler()

/opt/conda/envs/rapids/lib/python3.8/site-packages/dask_cloudprovider/azure/azurevm.py in create_vm(self)
73
74 async def create_vm(self):
---> 75 [subnet_info, *_] = await self.cluster.call_async(
76 self.cluster.network_client.subnets.list,
77 self.cluster.resource_group,

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/core/paging.py in __next__(self)
127 if self._page_iterator is None:
128 self._page_iterator = itertools.chain.from_iterable(self.by_page())
--> 129 return next(self._page_iterator)
130
131 next = __next__ # Python 2 compatibility.

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/core/paging.py in __next__(self)
74 raise StopIteration("End of paging")
75 try:
---> 76 self._response = self._get_next(self.continuation_token)
77 except AzureError as error:
78 if not error.continuation_token:

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/mgmt/network/v2020_06_01/operations/_subnets_operations.py in get_next(next_link)
659 request = prepare_request(next_link)
660
--> 661 pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
662 response = pipeline_response.http_response
663

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/core/pipeline/_base.py in run(self, request, **kwargs)
209 else _TransportRunner(self._transport)
210 )
--> 211 return first_node.send(pipeline_request) # type: ignore

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/core/pipeline/_base.py in send(self, request)
69 _await_result(self._policy.on_request, request)
70 try:
---> 71 response = self.next.send(request)
72 except Exception: # pylint: disable=broad-except
73 _await_result(self._policy.on_exception, request)

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/core/pipeline/_base.py in send(self, request)
69 _await_result(self._policy.on_request, request)
70 try:
---> 71 response = self.next.send(request)
72 except Exception: # pylint: disable=broad-except
73 _await_result(self._policy.on_exception, request)

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/core/pipeline/_base.py in send(self, request)
69 _await_result(self._policy.on_request, request)
70 try:
---> 71 response = self.next.send(request)
72 except Exception: # pylint: disable=broad-except
73 _await_result(self._policy.on_exception, request)

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/core/pipeline/_base.py in send(self, request)
69 _await_result(self._policy.on_request, request)
70 try:
---> 71 response = self.next.send(request)
72 except Exception: # pylint: disable=broad-except
73 _await_result(self._policy.on_exception, request)

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/core/pipeline/_base.py in send(self, request)
69 _await_result(self._policy.on_request, request)
70 try:
---> 71 response = self.next.send(request)
72 except Exception: # pylint: disable=broad-except
73 _await_result(self._policy.on_exception, request)

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/mgmt/core/policies/_base.py in send(self, request)
45 # type: (PipelineRequest[HTTPRequestType], Any) -> PipelineResponse[HTTPRequestType, HTTPResponseType]
46 http_request = request.http_request
---> 47 response = self.next.send(request)
48 if response.http_response.status_code == 409:
49 rp_name = self._check_rp_not_registered_err(response)

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/core/pipeline/policies/_redirect.py in send(self, request)
156 redirect_settings = self.configure_redirects(request.context.options)
157 while retryable:
--> 158 response = self.next.send(request)
159 redirect_location = self.get_redirect_location(response)
160 if redirect_location and redirect_settings['allow']:

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/core/pipeline/policies/_retry.py in send(self, request)
443 start_time = time.time()
444 self._configure_timeout(request, absolute_timeout, is_response_error)
--> 445 response = self.next.send(request)
446 if self.is_retry(retry_settings, response):
447 retry_active = self.increment(retry_settings, response=response)

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/core/pipeline/policies/_authentication.py in send(self, request)
115 :type request: ~azure.core.pipeline.PipelineRequest
116 """
--> 117 self.on_request(request)
118 try:
119 response = self.next.send(request)

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/core/pipeline/policies/_authentication.py in on_request(self, request)
92
93 if self._token is None or self._need_new_token:
---> 94 self._token = self._credential.get_token(*self._scopes)
95 self._update_headers(request.http_request.headers, self._token.token)
96

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/identity/_credentials/default.py in get_token(self, *scopes, **kwargs)
170 return token
171
--> 172 return super(DefaultAzureCredential, self).get_token(*scopes, **kwargs)

/opt/conda/envs/rapids/lib/python3.8/site-packages/azure/identity/_credentials/chained.py in get_token(self, *scopes, **kwargs)
106 "https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot."
107 _LOGGER.warning(message)
--> 108 raise ClientAuthenticationError(message=message)

ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot.this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
SharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
VisualStudioCodeCredential: Failed to get Azure user details from Visual Studio Code.
AzureCliCredential: Unexpected output from Azure CLI: 'bash: /opt/conda/envs/rapids/lib/libtinfo.so.6: no version information available (required by bash)
{
****,
"expiresOn": "2022-03-28 18:00:36.000000",
"subscription": "#####",
"tenant": "#####",
"tokenType": "Bearer"
}
'.
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/azclicredential/troubleshoot.
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
```
**Environment**:

- Dask version: 2022.3.0
- Dask Cloud Provider version: 2022.3.3
- Python version: 3.8.13
- Operating System: Ubuntu 20.04
- Install method (conda, pip, source): pip

Cluster Dump State:

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.