RedisConnectionException when trying to connect to AWS Redis cluster with multiple shards

Hello,

We are using an AWS ElastiCache Redis cluster as a cache mechanism, and we are using the ConnectionMultiplexer to create the connection as a singleton at the start of our application. This means that the number of open connections is low at any given time (way below the limit of AWS Redis).

Recently we started getting intermittent errors when connecting to the Redis cluster (during startup):

Unhandled exception. StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). UnableToConnect on clustercfg._redis-cluster-name_.3itdzk.euw1.cache.amazonaws.com:6379/Interactive, Initializing/NotStarted, last: NONE, origin: BeginConnectAsync, outstanding: 0, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, global: 0s ago, v: 2.2.4.27433

We don't get this exception all the time, and eventually, when we retry for a few minutes, we end up connecting and the application would behave normally (read and write operation would succeed without issue).

After some investigation, we noticed that this error only happens when we have more than 1 shard on our cluster, and we were able to reproduce this on our test environment by having a Redis cluster with 7 shards (each with 3 nodes (1 master and 2 replicas)).

We are connecting to the AWS ElastiCache Configuration Endpoint URL as per the AWS documentation.

After a few more tests, we also noticed that by setting the Redis client's ConfigurationOptions.Endpoints with just one of the shard's URLs (instead of using the ElastiCache Configuration URL), the error would no longer happen and the client would successfully connect all the time. Strangely enough, reads and writes would happen on all shards, not just the shard used on the endpoints configuration.

A few questions I would like to ask:

• Is Redis cluster mode supported by StackExchange.Redis client?

• Any idea why ConnectionMultiplexer would intermittently fail to connect when the cluster's configuration URL is used and multiple shards exist on the cluster?

• How does the StackExchange.Redis behave when configured with an ElastiCache Configuration URL? Does it become aware of all the shards and subsequently try to connect to each of them? Or does this routing happen at the server-side?

• What's the expected behavior of ConnectionMultiplexer when only the URLs of a specific shard are used to connect to? Is it expected that the client is still able to read/write from all other shards?

Thank you in advance,
Mário Pina

Start with ConnectionMultiplexer and ConfigurationOptions.Endpoints, then reproduce against an AWS ElastiCache Redis cluster with multiple shards using the configuration endpoint and a shard endpoint. Compare startup connection behavior and shard discovery, and consider the issue done when the intermittent failure is explained and a confirmed fix or expected behavior is documented.