Timeout awaiting response - random short streaks

Our .NET 8 application is running in Kubernetes, and we have 3 service pods using a Redis cache which is hosted as a dedicated redis server pod in the same Kubernetes namespace. The redis pod has a persisted storage and uses an appendonly file.

We're experiencing short streaks of redis client timeouts randomly throughout the day. For example, one day it's 20 failing requests within a period 10 seconds in the morning, the next day there's 10 failing requests within 5 seconds in the afternoon.

There are around 200k distributed cache requests per day, and even though the fail rate is very low (about 10-100 requests per day), they do have a negative effect on the application. The values stored and retrieved are very small, and hardly exceed a few kilobytes.

We do not see CPU spikes on either the redis client (application) or the redis server pod.

All exceptions have very similar metrics, and look as follows:

Timeout awaiting response (outbound=7KiB, inbound=0KiB, 5512ms elapsed, timeout is 5000ms), command=HSET, next: HSET b1547e0e-ca13-4531-ae31-3174bc7dea2b, inst: 0, qu: 0, qs: 1, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 7293, cur-in: 0, sync-ops: 0, async-ops: 24789596, serverEndpoint: redis-myserviceapi:6379, conn-sec: 2355933.28, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: myserviceapi-69b7b7ff56-njnds(SE.Redis-v2.7.17.27058), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=1,Free=32766,Min=1,Max=32767), POOL: (Threads=3,QueuedItems=0,CompletedItems=92320083,Timers=14), v: 2.7.17.27058 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

I cannot see anything suspicious here, although I must admit I find it hard to interpret these numbers.

The SLOWLOG from the redis server does not have any correlation with the client timeout exceptions.

How could we proceed in finding the root cause of (and mitigating action for) these timeouts? How can we even tell whether this is caused by the server or the client?

Start by interpreting the reported StackExchange.Redis timeout diagnostics, including the connection, queue, worker, and server endpoint fields. Compare those events with Redis SLOWLOG and the Kubernetes client and Redis pods to determine whether the delay is on the client or server side; done means identifying a likely root cause and a mitigation.