StackExchange / StackExchange/StackExchange.Redis

Sentinel +switch-master announcement missed if master dies while connected

Open
#2,487 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.2k
Forks
1.6k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

I'm testing Redis, StackExchange.Redis, and Redis-Sentinel in order to setup a appropriate HA environment.

I've noticed that when i stand up a sentinel cluster as described in these docs: https://hub.docker.com/r/bitnami/redis-sentinel/ and i shutdown the master after a multiplexer connection has been successfully established by a client using StackExchange.Redis. The multiplexer fails to realize that the failover has occurred and continues to queue messages for the now non-existent leader.

Funny thing is If i manually trigger a failover using the sentinel-cli with SENTINEL FAILOVER mymaster the multiplexer realizes the change and appropriately updates the leader.

I looked into the code and noticed that the +switch-master subscription hook is never fired when the primary Redis is shutdown even if the sentinel associated is still up and running. In the situation where i manually trigger the failover, the hook properly fires.

https://github.com/StackExchange/StackExchange.Redis/blob/f6171a19a0be078c6528b4631d42dfa4adcc8564/src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs#L33-L62

I've confirmed that the sentinel is still issuing the +switch-master command in both scenarios.

I've tried multiple ways to setup the multiplexer, here is my last iteration.

var opts = new ConfigurationOptions
            {
                SyncTimeout = 60000,
                AbortOnConnectFail = false,
                Ssl = false,
                Password = config.Password,
                ReconnectRetryPolicy = new LinearRetry(2000),
                ConnectRetry = 30,
                ServiceName = "mymaster",
                ConfigCheckSeconds = 10
            };
            opts.EndPoints.Add("redis-sentinel-1", 26379);
            opts.EndPoints.Add("redis-sentinel-2", 26379);
            opts.EndPoints.Add("redis-sentinel-3", 26379);

            var log = new LoggerWriter(_log, LogLevel.Information);

            //_multiplexer = await ConnectionMultiplexer.ConnectAsync(opts, log);
            _multiplexerSentinel = await ConnectionMultiplexer.SentinelConnectAsync(opts, log);
            _multiplexer = _multiplexerSentinel.GetSentinelMasterConnection(opts, log);


            _multiplexerSentinel.ConfigurationChanged += (s, e) =>
            {
                _log.LogError($"Sentinel configuration changed (Sentinel): {e.EndPoint}");
                _log.LogInformation($"New server status: {_multiplexerSentinel.GetStatus()}");
            };
            _multiplexerSentinel.ConnectionFailed += (s, e) =>
            {
                _log.LogError($"Connection failed (Sentinel): {e.ConnectionType}, {e.FailureType}, {e.EndPoint}, {e.Exception}");
                _log.LogInformation($"New server status: {_multiplexerSentinel.GetStatus()}");
            };
            _multiplexerSentinel.ConnectionRestored += (s, e) =>
            {
                _log.LogError($"Connection restored (Sentinel): {e.ConnectionType}, {e.FailureType}, {e.EndPoint}, {e.Exception}");
                _log.LogInformation($"New server status: {_multiplexerSentinel.GetStatus()}");
            };
            _multiplexerSentinel.ConfigurationChangedBroadcast += (s, e) =>
            {
                _log.LogError($"Connection changed via broadcast (Sentinel): {e.EndPoint}");
                _log.LogInformation($"New server status: {_multiplexerSentinel.GetStatus()}");
            };

            _multiplexer.ConfigurationChanged += (s, e) =>
            {
                _log.LogError($"Configuration changed: {e.EndPoint}");
                _log.LogInformation($"New server status: {_multiplexer.GetStatus()}");
                //var info = _multiplexer.GetServers().Where(x => x.IsConnected).Select(x => x.InfoRaw()).ToArray();
               // _multiplexer.Configure(new LoggerWriter(_log, LogLevel.Information));
                //_log.LogInformation($"Server info: {info}");
                // _multiplexer.PublishReconfigure();
            };
            _multiplexer.ConnectionFailed += (s, e) =>
            {
                _log.LogError($"Connection failed: {e.ConnectionType}, {e.FailureType}, {e.EndPoint}, {e.Exception}");
                _log.LogInformation($"New server status: {_multiplexer.GetStatus()}");
            };
            _multiplexer.ConnectionRestored += (s, e) =>
            {
                _log.LogError($"Connection restored: {e.ConnectionType}, {e.FailureType}, {e.EndPoint}, {e.Exception}");
                _log.LogInformation($"New server status: {_multiplexer.GetStatus()}");
            };
            _multiplexer.ConfigurationChangedBroadcast += (s, e) =>
            {
                _log.LogError($"Connection changed via broadcast: {e.EndPoint}");
                _log.LogInformation($"New server status: {_multiplexer.GetStatus()}");
            };

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs around the linked lines and reproduce the two failover paths described: shutting down the master and issuing SENTINEL FAILOVER mymaster. Trace why +switch-master is handled in one case but not the other; done means the multiplexer detects the shutdown-driven failover and updates the leader instead of queueing messages for the old master.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, redis
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.