StackExchange / StackExchange/StackExchange.Redis

Seamlessly handling Elasticache (AWS) failovers?

Open
#1,874 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

☁️ platform:AWS ⚙️ area:failover
Dominant language
C#
Stars
6.2k
Forks
1.6k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

Hi, is anyone else connecting to a Multi-AZ Elasticache cluster, and is able to seamlessly handle a failover in a way that makes the client block until the connection is reestablished, and then resumes? I'm using these options:

                var options = ConfigurationOptions.Parse(address);
                options.ReconnectRetryPolicy = new ExponentialRetry(250);
                options.SyncTimeout = 10000;
                options.AsyncTimeout = 10000;
                options.AbortOnConnectFail = false;

but when I initiate a failover, I immediately get this exception:

redis_connection_failed_message: Interactive#1@#####.usw2.cache.amazonaws.com:6379 (Idle)
redis_connection_failed_message: Subscription#2@#####.usw2.cache.amazonaws.com:6379 (Idle)

Unhandled Exception:
StackExchange.Redis.RedisConnectionException: No connection is active/available to service this operation: GET pubsub.test.value; SocketClosed (ReadEndOfStream, last-recv: 0) on #####.usw2.cache.amazonaws.com:6379/Subscription, Idle/MarkProcessed, last: SUBSCRIBE, origin: ReadFromPipe, outstanding: 0, last-read: 0s ago, last-write: 21s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 8 of 10 available, in: 0, in-pipe: 0, out-pipe: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.2.50.36290, mc: 1/1/0, mgr: 10 of 10 available, clientName: lax-jchow03-mac, IOCP: (Busy=0,Free=200,Min=16,Max=200), WORKER: (Busy=0,Free=1600,Min=16,Max=1600), v: 2.2.50.36290 ---> StackExchange.Redis.RedisConnectionException: SocketClosed (ReadEndOfStream, last-recv: 0) on #####.usw2.cache.amazonaws.com:6379/Subscription, Idle/MarkProcessed, last: SUBSCRIBE, origin: ReadFromPipe, outstanding: 0, last-read: 0s ago, last-write: 21s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 8 of 10 available, in: 0, in-pipe: 0, out-pipe: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.2.50.36290
--- End of inner exception stack trace ---

Is there another group of settings I'm missing? (I don't think 'connectRetry' is relevant)

Thanks in advance,
Jeff

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ConfigurationOptions settings and the reported RedisConnectionException during a Multi-AZ ElastiCache failover. Reproduce the failover with the shown options and review the connection and subscription behavior; done would require an agreed way for operations to resume after reconnection without the reported exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, csharp, redis
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.