Timeout error while performing INCR and MGET

Hi,

I am using StackExchange.Redis library in my web application. Following is the setup.

• Application is developed in .NET Framework 4.8 and ASP.NET MVC.
• Runs on Windows Server on AWS EC2.
• Connects to Redis Cluster on AWS ElastiCache.

Following are a few examples of errors.

Operation: MultiGet : KeyCount :2; Message: Timeout performing MGET (5000ms), next: MGET {SomeKey}:0449-0450, inst: 4, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: id-redis-rl-main-cache-0002-002.id-redis-rl-main-cache.u8dlwf.use1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: IP-0AC62558(SE.Redis-v2.6.66.47313), PerfCounterHelperkeyHashSlot: 12665, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=13,Free=32754,Min=8,Max=32767), v: 2.6.66.47313 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Operation: Increment; Message: Timeout performing INCRBYFLOAT (5000ms), next: INCRBYFLOAT {SomeKey}:0450-0451, inst: 4, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: id-redis-rl-main-cache-0002-001.id-redis-rl-main-cache.u8dlwf.use1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: IP-0AC62558(SE.Redis-v2.6.66.47313), PerfCounterHelperkeyHashSlot: 12665, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=20,Free=32747,Min=8,Max=32767), v: 2.6.66.47313 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Operation: Increment; Message: Timeout performing INCRBYFLOAT (5000ms), next: INCRBYFLOAT {somekey}:0614-0615, inst: 10, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: id-redis-rl-main-cache-0005-001.id-redis-rl-main-cache.my7vre.use1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: IP-0AC2B294(SE.Redis-v2.6.66.47313), PerfCounterHelperkeyHashSlot: 15309, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=26,Free=32741,Min=8,Max=32767), v: 2.6.66.47313 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Around 30 such errors reported at after application started to use Redis. Since then no timeout errors reported.
As of now I am using Redis only to perform Rate-Limit hence only INCR, EXPIRE and MGET are the only operations being performed. But I am planning to use the same Redis cluster to store business objects too which will have higher throughput of cache operations compared to the Rate-Limit usage.

I am really surprised to see the Timeout error for INCR and EXPIRE operations. Even I am using CommandFlags.FireAndForge to perform EPXPIRE after INCR.

var result = RedisDatabase.StringIncrement(key, delta, CommandFlags.FireAndForget); RedisDatabase.KeyExpire(key, validFor, CommandFlags.PreferMaster & CommandFlags.FireAndForget);

var result = RedisDatabase.StringGet(redisKeys, CommandFlags.PreferReplica);

I want to understand what could be the reason of timeout issues and fix them before I expand the usage of Redis for business object.

99% of the solutions online suggest to setup proper Worker Pool etc. But I wonder if that can be the only reason of this issue. I am interested to know what other factors can contribute to this such as Application throughput, EC2 instance size, Version of .NET Framework, size of cache key and/or cache data.

I will be also helpful to know if there is any formula or technique to know the optimal number of worker threads based on server size, number of cores, number of shards in the Redis cluster etc.

I would also like to understand why these simple operations take more than 2ms 30% of the times.

I am really looking forward to hints/approaches to understand and solve this issue. I can share more information about the issue I am facing if needed.

Thanks.

Start with the reported StringIncrement, KeyExpire, and StringGet calls and the linked StackExchange.Redis timeout guidance, then examine the captured timeout diagnostics and Redis cluster context. Done means identifying the contributing factor behind the INCRBYFLOAT, MGET, and related timeouts and documenting a supported mitigation before higher-throughput use.