StackExchange / StackExchange/StackExchange.Redis
Sentinel +switch-master announcement missed if master dies while connected
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.2k
- Forks
- 1.6k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 43
Description
I'm testing Redis, StackExchange.Redis, and Redis-Sentinel in order to setup a appropriate HA environment.
I've noticed that when i stand up a sentinel cluster as described in these docs: https://hub.docker.com/r/bitnami/redis-sentinel/ and i shutdown the master after a multiplexer connection has been successfully established by a client using StackExchange.Redis. The multiplexer fails to realize that the failover has occurred and continues to queue messages for the now non-existent leader.
Funny thing is If i manually trigger a failover using the sentinel-cli with SENTINEL FAILOVER mymaster the multiplexer realizes the change and appropriately updates the leader.
I looked into the code and noticed that the +switch-master subscription hook is never fired when the primary Redis is shutdown even if the sentinel associated is still up and running. In the situation where i manually trigger the failover, the hook properly fires.
I've confirmed that the sentinel is still issuing the +switch-master command in both scenarios.
I've tried multiple ways to setup the multiplexer, here is my last iteration.
var opts = new ConfigurationOptions
{
SyncTimeout = 60000,
AbortOnConnectFail = false,
Ssl = false,
Password = config.Password,
ReconnectRetryPolicy = new LinearRetry(2000),
ConnectRetry = 30,
ServiceName = "mymaster",
ConfigCheckSeconds = 10
};
opts.EndPoints.Add("redis-sentinel-1", 26379);
opts.EndPoints.Add("redis-sentinel-2", 26379);
opts.EndPoints.Add("redis-sentinel-3", 26379);
var log = new LoggerWriter(_log, LogLevel.Information);
//_multiplexer = await ConnectionMultiplexer.ConnectAsync(opts, log);
_multiplexerSentinel = await ConnectionMultiplexer.SentinelConnectAsync(opts, log);
_multiplexer = _multiplexerSentinel.GetSentinelMasterConnection(opts, log);
_multiplexerSentinel.ConfigurationChanged += (s, e) =>
{
_log.LogError($"Sentinel configuration changed (Sentinel): {e.EndPoint}");
_log.LogInformation($"New server status: {_multiplexerSentinel.GetStatus()}");
};
_multiplexerSentinel.ConnectionFailed += (s, e) =>
{
_log.LogError($"Connection failed (Sentinel): {e.ConnectionType}, {e.FailureType}, {e.EndPoint}, {e.Exception}");
_log.LogInformation($"New server status: {_multiplexerSentinel.GetStatus()}");
};
_multiplexerSentinel.ConnectionRestored += (s, e) =>
{
_log.LogError($"Connection restored (Sentinel): {e.ConnectionType}, {e.FailureType}, {e.EndPoint}, {e.Exception}");
_log.LogInformation($"New server status: {_multiplexerSentinel.GetStatus()}");
};
_multiplexerSentinel.ConfigurationChangedBroadcast += (s, e) =>
{
_log.LogError($"Connection changed via broadcast (Sentinel): {e.EndPoint}");
_log.LogInformation($"New server status: {_multiplexerSentinel.GetStatus()}");
};
_multiplexer.ConfigurationChanged += (s, e) =>
{
_log.LogError($"Configuration changed: {e.EndPoint}");
_log.LogInformation($"New server status: {_multiplexer.GetStatus()}");
//var info = _multiplexer.GetServers().Where(x => x.IsConnected).Select(x => x.InfoRaw()).ToArray();
// _multiplexer.Configure(new LoggerWriter(_log, LogLevel.Information));
//_log.LogInformation($"Server info: {info}");
// _multiplexer.PublishReconfigure();
};
_multiplexer.ConnectionFailed += (s, e) =>
{
_log.LogError($"Connection failed: {e.ConnectionType}, {e.FailureType}, {e.EndPoint}, {e.Exception}");
_log.LogInformation($"New server status: {_multiplexer.GetStatus()}");
};
_multiplexer.ConnectionRestored += (s, e) =>
{
_log.LogError($"Connection restored: {e.ConnectionType}, {e.FailureType}, {e.EndPoint}, {e.Exception}");
_log.LogInformation($"New server status: {_multiplexer.GetStatus()}");
};
_multiplexer.ConfigurationChangedBroadcast += (s, e) =>
{
_log.LogError($"Connection changed via broadcast: {e.EndPoint}");
_log.LogInformation($"New server status: {_multiplexer.GetStatus()}");
};
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/StackExchange.Redis/ConnectionMultiplexer.Sentinel.cs around the linked lines and reproduce the two failover paths described: shutting down the master and issuing SENTINEL FAILOVER mymaster. Trace why +switch-master is handled in one case but not the other; done means the multiplexer detects the shutdown-driven failover and updates the leader instead of queueing messages for the old master.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, redis
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100