Timeout Exceptions in .NET 6 (Azure Redis Cache + Kubernetes)

Hello!

We have a .NET Core MVC Application that is deployed to multiple K8s pods, all using the same Azure Redis Cache instance. We are currently experiencing a large number of timeout exceptions, and have been trying to diagnose them for weeks with no luck. A few exception examples:

Timeout performing GET (5000ms), next: GET {KEY}, inst: 0, qu: 0, qs: 1, aw: False, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: {ENDPOINT NAME}, mc: 1/1/0, mgr: 10 of 10 available, clientName: {CLIENTNAME}, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=1,Free=32766,Min=10,Max=32767), v: 2.2.4.27433 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Timeout performing SETEX (5000ms), next: SETEX {KEY}, inst: 0, qu: 0, qs: 1, aw: False, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: {ENDPOINT NAME}, mc: 1/1/0, mgr: 10 of 10 available, clientName: {CLIENTNAME}, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=1,Free=32766,Min=10,Max=32767), v: 2.2.4.27433 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

We are using a ConnectionMultiplexer that is registered as a singleton, and is then used in our CacheService class:
public string GetCacheString(string key) { var cache = _connectionMultiplexer.GetDatabase(); return cache.StringGet(key); }

So far we have tried the following with no luck:

• Using the IDistributedCache interface
• Increasing thread count

Our SLOWLOG does report slow commands, but none of them are taking longer than the timeout value we have set (default 5000ms). A note on our apps behavior - there can be thousands of users hitting our app at once, and most requests utilize Redis in some way to see if data exists in Redis before querying our SQL DB. These users can be sending the same exact requests at the same time.

Any help with this is appreciated!!

Start with the reported ConnectionMultiplexer singleton and CacheService.GetCacheString entry point, then review the timeout diagnostics alongside the Azure Redis Cache and Kubernetes deployment details. Done would require identifying the cause of the GET and SETEX timeouts and documenting or implementing a confirmed fix, but the issue does not name files or tests.