Azure / Azure/azure-functions-host

[BUG] azure function ManagedIdentity request exception during GetToken

Open
#11,058 6 comments 2 reactions 0 assignees View on GitHub
Needs: Triage (Functions)
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

# Issue Transfer

This issue has been transferred from the Azure SDK for .NET repository, [#49912](https://github.com/Azure/azure-sdk-for-net/issues/49912).

### Please be aware that @patrick-hofer is the author of the original issue and include them for any questions or replies.

## Azure SDK Analysis

This is expected behavior of the credential when token acquisition fails. In this case, the cause is a failure with the local MI endpoint in the application host machine. The Azure.Identity credential only makes an HTTP request according to the MI contract and has no insight nor influence over the availability or responsiveness of the endpoint.

Investigation by the team that owns the host machine environment is needed.

## Details

### Describe the bug

Since 28. April some of our azure functions suddenly started to get the following exception when calling the system-assigned ManagedIdentity getToken endpoint using the ManagedIdentityCredentials class.

**ManagedIdentityCredential authentication unavailable. Multiple attempts failed to obtain a token from the managed identity endpoint.**

The error does not occur always, in our development environment it occurs in 1 out of 10 requests.
Playing around with the retry policy (as suggested by the error message) did not fix the problem.

`Azure.Identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable. Multiple attempts failed to obtain a token from the managed identity endpoint.
---> MSAL.NetCore.4.67.2.0.MsalServiceException:
ErrorCode: managed_identity_request_failed
Microsoft.Identity.Client.MsalServiceException: Retry failed after 6 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80))
---> System.AggregateException: Retry failed after 6 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80))
---> Azure.RequestFailedException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)
---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)
---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.HttpClientTransport.ProcessSyncOrAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory1 pipeline, Boolean async)
at Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
at Azure.Core.HttpPipelineMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Microsoft.Identity.Client.Http.HttpManager.ExecuteAsync(Uri endpoint, IDictionary2 headers, HttpContent body, HttpMethod method, X509Certificate2 bindingCertificate, HttpClient customHttpClient, ILoggerAdapter logger, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Http.HttpManager.SendRequestAsync(Uri endpoint, IDictionary2 headers, HttpContent body, HttpMethod method, ILoggerAdapter logger, Boolean doNotThrow, X509Certificate2 bindingCertificate, HttpClient customHttpClient, CancellationToken cancellationToken, Int32 retryCount)
at Microsoft.Identity.Client.ManagedIdentity.AbstractManagedIdentity.AuthenticateAsync(AcquireTokenForManagedIdentityParameters parameters, CancellationToken cancellationToken)
---> (Inner Exception #1) Azure.RequestFailedException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)
---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)
---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpC...`

Multiple azure functions that are untouched since multiple months show this behaviour as well as newly created ones.

The error looks like this in azure application insights:
![Image](https://github.com/user-attachments/assets/41be73e4-48a0-405a-9376-a0469bb68f0a)

### Expected behavior

ManagedIdentityCredential should successfully get the token and proceed with execution.

### Actual behavior

The token cant be aquired leading to an exception and a failed operation.

### Reproduction Steps

We use Azure.DigitalTwins.Core 1.4.0 to get data from the Azure Digital Twins service.
The DigitalTwinsClient is registered as singleton and gets an instance of ManagedIdentityCredential as TokenCredential
`new DigitalTwinsClient(new Uri(url), new ManagedIdentityCredential(), ... `
Later on the client.GetDigitalTwin method is called which triggers the getToken

### Environment

Details regarding our DEV environment:

Azure function(s):
.net 8 isolated
32bit
using system assigned managed identity

OS: Windows
Location: West Europe
Runtime version: 4.1038.300.25164

App service plan:
Pricing: Y1
Zone redundant: Disabled
Consumption

Contributor guide

Open the contributing guide

Research direction

No repository file or test is identified in the issue. Start by reviewing the Azure SDK Analysis and the managed identity endpoint failure details; the issue is only actionable once the responsible host-environment investigation identifies a change within Azure Functions Host and a reproducible validation path.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
authentication, backend, cloud
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.