Azure / Azure/azure-functions-host

ManagedIdentityCredential authentication unavailable

Open
#10,189 1 comment 1 reaction 0 assignees View on GitHub
potential-bug
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

# Issue Transfer

This issue has been transferred from the Azure SDK for .NET repository, [#44294](https://github.com/Azure/azure-sdk-for-net/issues/44294).

### Please be aware that @dnmalenke is the author of the original issue and include them for any questions or replies.

## Details

have a simple Azure function to grab a couple secrets from keyvault on a timer trigger. If I restart the application and manually trigger the function, it works great, but if I leave it and wait for the time trigger, it will fail with this exception:

```
Result: Failure
Exception: Azure.Identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable. Multiple attempts failed to obtain a token from the managed identity endpoint.
---> System.AggregateException: Retry failed after 6 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)) (An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80))
---> Azure.RequestFailedException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)
---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)
---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
at Azure.Identity.ImdsManagedIdentitySource.AuthenticateAsync(Boolean async, TokenRequestContext context, CancellationToken cancellationToken)
---> (Inner Exception #1) Azure.RequestFailedException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)
---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions. (169.254.169.254:80)
---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
--- End of inner exception stack trace ---
at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
at Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
at Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)<---
```

### Expected behavior

ManagedIdentityCredential should authenticate with the given system assigned identity.

### Actual behavior

The function works sporadically. It usually always works on the first run after a restart. After that, it seems to not be reliable.

I have also tried to initialize the SecretClient with the ManagedIdentityCredential in the constructor of the class, but get the same results.

### Reproduction Steps

```
public class TimerTrigger1
{
private readonly ILogger _logger;

public TimerTrigger1(ILoggerFactory loggerFactory)
{
_logger = loggerFactory.CreateLogger();
}

[Function("TimerTrigger1")]
public void Run([TimerTrigger("0 0 7 * * *")] TimerInfo myTimer)
{
_logger.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
var secretClient = new SecretClient(new Uri("https://XXXXX.vault.azure.net/"), new ManagedIdentityCredential());
}
}
```

### Environment

I manually configured the time zone via environment variable:
WEBSITE_TIME_ZONE: Eastern Standard Time

Function runtime version: ~4
.Net Version: .NET 8 isolated
64 bit Windows
No networking options have been configured
System assigned identity is enabled and given proper Key Vault role assignments

Contributor guide

Open the contributing guide

Research direction

Start with the TimerTrigger1 reproduction and the reported ManagedIdentityCredential failure against 169.254.169.254 on .NET 8 isolated, Windows, and Functions runtime ~4. No repository file or test is named in the report. Done means identifying and correcting the sporadic authentication behavior so scheduled executions consistently authenticate with the system-assigned identity.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.