Azure / Azure/Microsoft.Azure.StackExchangeRedis

AcquireTokenAsync awaits TokenRefreshed/TokenRefreshFailed handlers without ConfigureAwait(false), deadlocking on ASP.NET when the token is cache-served

Open Beginner friendly
#95 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
21
Forks
24
PR merge metrics
No merged PRs in 30d

Description

### Summary

`AzureCacheOptionsProviderWithToken.AcquireTokenAsync` awaits `InvokeHandlerWithTimeoutAsync` **without `ConfigureAwait(false)`** in two places. On any platform with a non-default `SynchronizationContext` — classic ASP.NET being the important one, which this repo ships samples for — this deadlocks permanently if the caller blocks on the returned task.

In [`src/AzureCacheOptionsProviderWithToken.cs`](https://github.com/Azure/Microsoft.Azure.StackExchangeRedis/blob/main/src/AzureCacheOptionsProviderWithToken.cs) on `main`:

```csharp
// line 230 — token-failure path
await InvokeHandlerWithTimeoutAsync(() => TokenRefreshFailed?.Invoke(this, new(lastException, _tokenExpiry)), nameof(TokenRefreshFailed));

// line 258 — token-success path
await InvokeHandlerWithTimeoutAsync(() => TokenRefreshed?.Invoke(this, tokenResult), nameof(TokenRefreshed));
```

This looks like an oversight rather than a decision: the file has **15 awaits, and 13 of them use `ConfigureAwait(false)`**. The only two that don't are the pair above. The *same helper* is awaited with `ConfigureAwait(false)` three times elsewhere in the same file (lines 314, 326, 345 in `ReauthenticateConnectionsAsync`), and `InvokeHandlerWithTimeoutAsync`'s own internal await has it (line 374).

### Why this deadlocks

The subtle part is that it only bites when the token acquisition **succeeds quickly**.

`ConfigureForAzureWithTokenCredentialAsync` → `ConfigureForAzureAsync` → `AcquireTokenAsync` contains no suspension point before line 258 when `Azure.Identity` serves the token from its in-process cache. Nothing awaits asynchronously, so execution is still on the **caller's** thread with the caller's `SynchronizationContext` current. The bare await at line 258 therefore captures it and posts the continuation there.

Under classic ASP.NET, `AspNetSynchronizationContext` serialises posted callbacks behind the request currently executing. If that request is the one blocked waiting for the configuration task, the continuation is queued **behind its own waiter** and can never run.

A cache miss masks the bug: the real HTTP round-trip suspends, the method resumes on a thread-pool thread with no ambient context, and line 258 captures nothing. So this reproduces intermittently and, counter-intuitively, **only when the identity provider is healthy and fast**.

### Impact

We hit this in production. A host wedged for 27+ minutes until restarted; it does not self-recover. From a memory dump of the wedged process:

- The token **had** been acquired successfully 27 minutes earlier — `_token` and `_user` were populated.
- The awaited task was `RanToCompletion`.
- Every thread-pool queue was empty and no thread was executing the continuation (so: not thread-pool starvation — `MinThreads` was 1000, CPU ~9%).
- 384 of 441 managed threads were piled up behind the locks held by the one blocked caller.
- A reverse-reference walk from the stuck `d__34` state machine (`<>1__state = 3`, awaiter slot `<>u__3` a **bare `TaskAwaiter`**, versus `<>u__1`/`<>u__2` which are `ConfiguredTaskAwaiter`) resolved to:

```
d__34
-> AsyncMethodBuilderCore+MoveNextRunner
-> System.Action
-> System.Web.AspNetSynchronizationContext+<>c__DisplayClass22_0
-> System.Action
-> System.Web.Util.SynchronizationHelper+<>c__DisplayClass22_0
-> System.Action
-> ContinuationTaskFromTask
-> StandardTaskContinuation + System.Web.Util.SynchronizationHelper
```

That is the continuation sitting in the ASP.NET context's queue, behind the request that is blocked waiting for it.

The blocking caller is ours, and we accept that sync-over-async is our problem to avoid. But the library makes it unavoidable for anyone in this situation: the capture happens on the caller's thread before control ever returns, so a consumer has no way to opt out short of clearing `SynchronizationContext.Current` around the call — which is what we ended up doing as a workaround.

### Repro

1. Classic ASP.NET (.NET Framework), so `SynchronizationContext.Current` is `AspNetSynchronizationContext`.
2. From a request thread, call `ConfigurationOptions.ConfigureForAzureWithTokenCredentialAsync(credential)` and block on the result (`.GetAwaiter().GetResult()` / `.Result` / `.Wait()`).
3. Ensure the token is served from `Azure.Identity`'s cache so nothing in the call graph suspends — e.g. make a successful call first, then repeat.

Result: the second call never returns.

A `SynchronizationContext` whose `Post` queues behind the current operation is sufficient to reproduce outside ASP.NET; a plain thread-pool-dispatching context will not show it.

### Suggested fix

Add `.ConfigureAwait(false)` to both call sites, matching the other 13 awaits in the file:

```csharp
await InvokeHandlerWithTimeoutAsync(..., nameof(TokenRefreshFailed)).ConfigureAwait(false);
await InvokeHandlerWithTimeoutAsync(..., nameof(TokenRefreshed)).ConfigureAwait(false);
```

Worth considering as a follow-up: a `.editorconfig` rule or the `ConfigureAwaitChecker`/CA2007 analyzer on this project would prevent recurrence, since the library is explicitly consumed from `SynchronizationContext`-bearing hosts (see `ASP.NET_Samples/ASP.NET_Framework/`).

### Environment

| | |
|---|---|
| Package | `Microsoft.Azure.StackExchangeRedis` 3.3.1 (current latest) |
| Also present on | `main` at time of filing |
| Runtime | .NET Framework 4.8, classic ASP.NET (System.Web) |
| Auth | Managed Identity via `ManagedIdentityCredential` |

Verified against both the shipped 3.3.1 assembly (decompiled) and the current `main` source.

Contributor guide

Open the contributing guide

Research direction

Start in src/AzureCacheOptionsProviderWithToken.cs at AcquireTokenAsync and inspect the two InvokeHandlerWithTimeoutAsync awaits described in the issue, comparing them with the other awaits in the file. Verify the change with the classic ASP.NET reproduction using a cache-served token; done means the blocked caller returns instead of deadlocking.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
authentication
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
85/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.