Excessive HELLO on stable connections under ConfigureForAzureWithTokenCredentialAsync (Entra token) — ~1 HELLO/connection/second, ~11–22% of all commands

**Prepared for submission to** `Azure/Microsoft.Azure.StackExchangeRedis` (and possibly `StackExchange/StackExchange.Redis`).
**Date:** 2026-07-06.

## Summary

When a `ConnectionMultiplexer` authenticates to **Azure Managed Redis** with
`ConfigurationOptions.ConfigureForAzureWithTokenCredentialAsync(TokenCredential)` (Entra ID / managed identity),
the client re-issues the **full `HELLO` handshake on stable, long-lived connections at a rate of ~1 per
connection per second**. On a busy client this is **~11–22% of all commands** sent to the server
(`cmdstat_hello`). The **identical** connection authenticated with a **static access-key password** does **not**
exhibit this — `HELLO` occurs only at connection establishment.

This appears to be a defect: a persistent connection should send `HELLO` **once** (at handshake), and token
**re-authentication** should be an `AUTH` command near token expiry (which the extension does implement). We
observe neither expiry-driven nor `AUTH`-based re-auth — instead a continuous stream of `HELLO` on connections
that are hours old and never dropped.

## Environment

| | |
|---|---|
| `Microsoft.Azure.StackExchangeRedis` | **3.3.1** (latest) |
| `StackExchange.Redis` | **2.13.17** |
| `Azure.Identity` | 1.17.1 |
| Runtime | .NET 8 / .NET 10, Linux (Azure Container Apps) |
| Cache | **Azure Managed Redis**, Balanced B20, **OSS clustering policy**, port 10000, TLS, Redis 7.4 |
| Auth | Microsoft Entra ID via managed identity (`DefaultAzureCredential`); access keys normally disabled |
| Protocol | Reproduced on **both RESP2 and RESP3** |

## Expected behavior

Per the Microsoft Learn guidance ("Use Microsoft Entra for cache authentication with Azure Managed Redis"), the
extension holds persistent connections and **proactively re-authenticates ~before token expiry** (token lifetime
~hours). A persistent connection should therefore emit:
- `HELLO` **once** per physical connection (handshake), and
- `AUTH` **occasionally** (near token expiry, ~hourly), for in-place re-auth.

## Actual behavior

- `cmdstat_hello` grows at **~1 × (physical connection count) per second**, indefinitely, on connections that are
**not** being dropped/recreated.
- `cmdstat_auth` is **0** — the documented `AUTH`-based re-auth is not what's firing.
- Swapping **only** the credential to a static access-key password drops `cmdstat_hello` to ~0 (handshake only).

### Measurements (our production-shaped system, single-shard B20)

- **A/B across a fleet of clients, all else constant** (RESP2, same hosts, idle ~550 cmd/s):
- Entra token: **`cmdstat_hello` ≈ 77/s** (~11% of commands at idle; **~21% under load**), `cmdstat_auth` 0/s, new connections ~1.5/s, connection `age` = **hours**.
- Static access key: **`cmdstat_hello` ≈ 0.1/s**, `cmdstat_auth` 0/s.
- **Minimal single-client reproducer** (shared cache, low baseline): Entra token **5.14 HELLO/s** vs key **0.92 HELLO/s** (the 0.92 is shared-cache baseline + this client's initial handshake; on a dedicated idle cache the key run is ~0). One `ConnectionMultiplexer` here holds ~4 physical connections (OSS-cluster endpoint + shard node, × interactive+subscription on RESP2), i.e. ~1.25 HELLO/connection/s under the token credential.

### What we ruled out (evidence)

1. **Not token expiry.** The acquired token is valid ~24h; the extension logs `Acquired token with expiration: <+24h>` and there are **zero** `"Current token expired"` warnings while HELLO churns.
2. **Not the documented re-auth.** `AzureCacheOptionsProviderWithToken.ReauthenticateConnectionsAsync` issues `server.ExecuteAsync("AUTH", [User, Password])` (i.e. re-auth is `AUTH`, not `HELLO`), and it only runs on the ~2-minute heartbeat when the token is within ~5 min of expiry. `cmdstat_auth` measured **0/s** — consistent with a healthy 24h token, and confirming this path is **not** the source of the HELLO.
3. **Not reconnects.** `total_connections_received` grows ~1.5/s while HELLO is ~77/s; connections in `CLIENT LIST` are hours old (`age`).
4. **Not the SE.Redis heartbeat.** `cmdstat_ping` ≈ 1.5/s (≈ 1 per connection per ~46s); HELLO is ~46× that.
5. **The HELLO is AUTH-bearing.** `MONITOR` shows **0** HELLO during a window in which `cmdstat_hello` increased by 428 (over 6s) on the same node — i.e. Redis is redacting the command as it does for `AUTH`. So the churn is `HELLO … AUTH` (a full handshake carrying credentials), repeatedly, on established connections.

## Minimal reproduction

A self-contained reproducer (no product dependencies) is attached: `HelloRepro` (Program.cs + HelloRepro.csproj,
~120 lines; `README.md` documents the protocol). Against an otherwise-idle Azure Managed Redis:

```bash
# Entra token (managed identity in Azure; 'cli' locally after `az login`)
dotnet run -c Release -- --endpoint ..redis.azure.net:10000 --auth mi --seconds 60
# static access key
dotnet run -c Release -- --endpoint ..redis.azure.net:10000 --auth key --key --seconds 60
```

It connects one `ConnectionMultiplexer`, keeps it lightly active (one `GET`/s), and reports the server's
`cmdstat_hello` / `cmdstat_auth` delta rate. Expected: the `mi` run shows many HELLO/s (~1 × connection count),
the `key` run ~0. (Reproduces on both `--protocol resp2` and `resp3`.)

## Impact

On a single-shard cache, HELLO consuming ~11–22% of the per-node command budget is the **binding throughput
constraint** — it directly caps achievable ops/s. Our only mitigation today is to authenticate that connection
with a static access key (which reverses the recommended passwordless/managed-identity posture and requires key
rotation), or to accept the ~22% tax.

## Questions / ask

1. Is `HELLO` being re-issued on established connections (~1/connection/s) under a `TokenCredential`-backed
`DefaultOptionsProvider` **expected**? If so, what drives that cadence, and is it configurable?
2. If not expected: the handshake should occur once per physical connection, and token refresh should use `AUTH`
near expiry (as `ReauthenticateConnectionsAsync` already does) — the continuous `HELLO` looks like the client
re-running the full handshake unnecessarily. Could this be a `StackExchange.Redis` connection-maintenance
interaction with the dynamic `DefaultOptionsProvider` (`User`/`Password` re-read → re-handshake)?
3. Is there a supported way to keep managed-identity auth **and** avoid the per-connection HELLO churn (e.g. a
fix, a configuration flag, or a newer version)? `3.3.1` is the latest on NuGet as of this writing.

Happy to provide `CLIENT LIST`/`INFO commandstats` captures, the full investigation notes, or run the reproducer
against a build with a candidate fix.

Reproduction:

`// HelloRepro — minimal reproducer + "20%-HELLO test protocol" for the excessive-HELLO issue on
// Azure Managed Redis when authenticating with Microsoft.Azure.StackExchangeRedis (Entra token credential).
//
// SYMPTOM: a client authenticated via ConfigureForAzureWithTokenCredentialAsync(TokenCredential) causes
// StackExchange.Redis to re-issue the full HELLO handshake on STABLE, long-lived connections at ~1 per
// connection per second — showing up as ~11-22% of all commands to the server (cmdstat_hello). The SAME
// connection authenticated with a static access-key password emits HELLO only at connection-establishment
// (~0/s steady). It is NOT token expiry (a healthy multi-hour token) and NOT the documented Az.SE.Redis
// re-auth (that is an AUTH command; cmdstat_auth stays 0).
//
// PROTOCOL: run this against an OTHERWISE-IDLE cache twice, same everything except --auth, and compare
// cmdstat_hello/s:
// dotnet run -- --endpoint ..redis.azure.net:10000 --auth mi
// dotnet run -- --endpoint ..redis.azure.net:10000 --auth key --key
// Buggy library => mi HELLO/s >> key HELLO/s (mi churns ~1/conn/s; key ~0).
// FIXED library => mi HELLO/s ≈ key HELLO/s ≈ ~0 (only the initial handshake). <-- the gate for RD-022.
//
// --auth values: mi (DefaultAzureCredential) | cli (AzureCliCredential, for local az-login) | key (static key)
// Prereqs for mi/cli: the identity must be a Redis Data user on the cache. For key: access-key auth enabled.

using Azure.Identity;
using Microsoft.Extensions.Logging;
using StackExchange.Redis;
using System.Text.RegularExpressions;

string? endpoint = null, key = null;
string auth = "mi", protocol = "resp2";
int seconds = 60;
for (int i = 0; i < args.Length; i++)
{
switch (args[i])
{
case "--endpoint": endpoint = args[++i]; break;
case "--auth": auth = args[++i]; break;
case "--key": key = args[++i]; break;
case "--protocol": protocol = args[++i]; break;
case "--seconds": seconds = int.Parse(args[++i]); break;
default: Console.Error.WriteLine($"unknown arg: {args[i]}"); return 2;
}
}
if (string.IsNullOrWhiteSpace(endpoint))
{
Console.Error.WriteLine("required: --endpoint [--auth mi|cli|key] [--key ] [--protocol resp2|resp3] [--seconds N]");
return 2;
}

using var loggerFactory = LoggerFactory.Create(b =>
b.AddSimpleConsole(o => { o.SingleLine = true; o.TimestampFormat = "HH:mm:ss "; })
.SetMinimumLevel(LogLevel.Information));

var cfg = ConfigurationOptions.Parse(endpoint);
cfg.AbortOnConnectFail = false;
cfg.ClientName = $"hello-repro-{auth}";
cfg.LoggerFactory = loggerFactory; // surfaces Az.SE.Redis token-lifecycle logs (proves the token is healthy, not expiring)
cfg.Protocol = protocol.Equals("resp3", StringComparison.OrdinalIgnoreCase) ? RedisProtocol.Resp3 : RedisProtocol.Resp2;

switch (auth.ToLowerInvariant())
{
case "mi": await cfg.ConfigureForAzureWithTokenCredentialAsync(new DefaultAzureCredential()); break;
case "cli": await cfg.ConfigureForAzureWithTokenCredentialAsync(new AzureCliCredential()); break;
case "key":
if (string.IsNullOrWhiteSpace(key)) { Console.Error.WriteLine("--key required for --auth key"); return 2; }
cfg.Password = key; // static access-key auth (default AMR user)
break;
default: Console.Error.WriteLine($"unknown --auth '{auth}' (use mi|cli|key)"); return 2;
}

Console.WriteLine($"connecting endpoint={endpoint} auth={auth} protocol={cfg.Protocol} window={seconds}s");
await using var mux = await ConnectionMultiplexer.ConnectAsync(cfg);
var db = mux.GetDatabase();
await db.PingAsync();
Console.WriteLine($"connected endpoints={mux.GetEndPoints().Length} (cmdstat_hello is GLOBAL — run on an idle cache so deltas are attributable to this client)");

long Calls(string cmd)
{
long total = 0;
foreach (var ep in mux.GetEndPoints())
{
var server = mux.GetServer(ep);
if (!server.IsConnected) continue;
var info = (string?)(server.Execute("INFO", "commandstats")) ?? "";
foreach (var line in info.Split('\n'))
if (line.StartsWith($"cmdstat_{cmd}:", StringComparison.Ordinal))
{
var m = Regex.Match(line, @"calls=(\d+)");
if (m.Success) total += long.Parse(m.Groups[1].Value);
}
}
return total;
}

long hello0 = Calls("hello"), auth0 = Calls("auth");
var sw = System.Diagnostics.Stopwatch.StartNew();
// Keep the connection lightly active, as a real app would (idle connections still churn HELLO; light traffic
// makes the per-activity re-handshake visible without dominating cmdstat).
while (sw.Elapsed.TotalSeconds < seconds)
{
await db.StringGetAsync("hello-repro:probe");
await Task.Delay(1000);
}
sw.Stop();
long hello1 = Calls("hello"), auth1 = Calls("auth");
double w = sw.Elapsed.TotalSeconds;

Console.WriteLine();
Console.WriteLine($"=== RESULT auth={auth} protocol={cfg.Protocol} window={w:N0}s ===");
Console.WriteLine($" cmdstat_hello : +{hello1 - hello0,-7} = {(hello1 - hello0) / w,7:N2}/s");
Console.WriteLine($" cmdstat_auth : +{auth1 - auth0,-7} = {(auth1 - auth0) / w,7:N2}/s");
Console.WriteLine();
Console.WriteLine(" Interpretation:");
Console.WriteLine(" - BUGGY library: --auth mi/cli shows HELLO ~1/connection/s (many/s); --auth key shows ~0/s.");
Console.WriteLine(" - FIXED library: --auth mi/cli HELLO/s falls to ~key's (~0/s, only the initial handshake).");
Console.WriteLine(" - cmdstat_auth should be ~0 in all cases (Az.SE.Redis re-auth is AUTH and only fires near token expiry).");
return 0;
`

Start with the attached HelloRepro files, especially Program.cs, HelloRepro.csproj, and README.md, and run the documented managed-identity versus access-key commands on an otherwise idle cache. Trace ConfigureForAzureWithTokenCredentialAsync and ReauthenticateConnectionsAsync, then compare cmdstat_hello and cmdstat_auth; done means managed-identity authentication no longer produces continuous HELLO churn on stable connections.

Written by the indexing model from the issue text.