Timeout awaiting response

Hi,

I would like to ask about some advice regarding Redis timeouts. I'll try to describe the issue here:

We've encountering timeouts on Redis, but not all the time, happening each 4 hours, we think it's about we have batch jobs adding stuff to cache at those times, but still that's suspicious since those jobs are running all the time, but likely at those times there is a bit more load.

We have another app that uses the cache quite heavily, and records those redis timeouts, not many, and only around those particular times (each 4 hours, as mentioned).

We tried different things suggested mostly in the article from the error message (https://stackexchange.github.io/StackExchange.Redis/Timeouts)

We've introduced the connection multiplexer pooling, separated the connection multiplexer pool used by the endpoint that times out, so it doesn't interfere with another endpoint that's used by the batch job. Finally changed the MIN threads on the ThreadPool to 10, since the pod's have just 1 cpu, so it was MIN=1 by default.

But none of those got rid of the timeouts, althought there seem to be some improvements in terms of Redis latency, or on some parameters which are on the Timeout Exception (like qs is now quite low).

Do you see anything that can be improved based on those exceptions ?

This is one of sample Exceptions before we did any 'improvements':
Stack trace
Error message:
Timeout awaiting response (outbound=188KiB, inbound=0KiB, 5040ms elapsed, timeout is 5000ms), command=MGET, next: MSET, inst: 0, qu: 0, qs: 705, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 0, async-ops: 40737620, serverEndpoint: , conn-sec: 213442.35, mc: 1/1/0, mgr: 10 of 10 available, clientName: (SE.Redis-v2.6.90.64945), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=2,Free=32765,Min=1,Max=32767), POOL: (Threads=5,QueuedItems=0,CompletedItems=487697475), v: 2.6.90.64945 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Most recent error messages:
Stack trace
Error message:
Timeout awaiting response (outbound=45KiB, inbound=0KiB, 5220ms elapsed, timeout is 5000ms), command=MGET, next: MGET , inst: 0, qu: 0, qs: 123, aw: False, bw: Inactive, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 178, cur-in: 0, sync-ops: 0, async-ops: 285, serverEndpoint: , conn-sec: 634.02, mc: 1/1/0, mgr: 9 of 10 available, clientName: (SE.Redis-v2.6.90.64945), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=4,Free=32763,Min=10,Max=32767), POOL: (Threads=15,QueuedItems=1,CompletedItems=1339961), v: 2.6.90.64945 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Stack trace
Error message:
Timeout awaiting response (outbound=59KiB, inbound=0KiB, 5450ms elapsed, timeout is 5000ms), command=MGET, next: MGET, inst: 0, qu: 0, qs: 111, aw: False, bw: Inactive, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 0, cur-in: 0, sync-ops: 0, async-ops: 22695, serverEndpoint: , conn-sec: 746.1, mc: 1/1/0, mgr: 10 of 10 available, clientName: (SE.Redis-v2.6.90.64945), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=3,Free=32764,Min=10,Max=32767), POOL: (Threads=17,QueuedItems=0,CompletedItems=1600796), v: 2.6.90.64945 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Thank you for any hints/suggestions that might help to resolve this.

Start with the linked StackExchange.Redis timeout guidance and compare the earlier and recent exception details, especially the recurring MGET/MSET timeouts and queue values. Check the reported batch-job timing, connection pooling, and thread-pool changes; done would require identifying a reproducible cause or a specific client-side improvement supported by the exception data.