StackExchange / StackExchange/StackExchange.Redis

Abnormal master-slave switching in sentinelmode

Open
#2,841 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.2k
Forks
1.6k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

Thank you very much for your contribution. I encountered some difficulties when using it and would like to ask for advice.

My redis configuration is 1 master A, 1 slave B, 3 sentinels.

In sentinel mode, I found that after node A went offline, node B was upgraded to the master node.
There was no problem in this process.

But when the old master node A comes online again, stackExchange.redis will briefly mark both nodes as primary, and use the old master node A.
But A is converting to a slave node at this time and synchronizing new data. This will lead to the error of reading and writing A at this time.

The program will mark A as a replica later, and the error will be corrected.

Regarding this problem, how should I solve it? Is there any relevant configuration that can solve this problem?

MyTestCode

using StackExchange.Redis;
using static StackExchange.Redis.Role;

namespace ConsoleApp3
{
    internal class Program
    {
        static async Task Main(string[] args)
        {
            Console.WriteLine("Hello, World!");

            ConnectionMultiplexer masterConnection = ConnectionMultiplexer.Connect("172.16.184.197:26379,172.16.184.198:26379,172.16.0.29:26379,serviceName=mymaster");
            while (true)
            {
                try
                {
                    IDatabase database = masterConnection.GetDatabase();
                    await database.StringIncrementAsync("SomeKey", 1);
                    System.Console.WriteLine(masterConnection.GetStatus());
                    System.Console.WriteLine(DateTime.Now.ToString("yyyyMMddHHmmss::") + await database.StringGetAsync("SomeKey"));
                }
                catch (Exception ex)
                {
                    Console.WriteLine(ex.Message);
                }
                await Task.Delay(1000);
            }
        }
    }
}

Start

Image

stop A and switch to B

Image

start A and the data is dirty

Image

recover normal

Image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Use the C# sample's ConnectionMultiplexer.Connect(...) with serviceName=mymaster and the three Sentinel endpoints to reproduce the failover. Start by tracing how Sentinel role changes are observed after A returns and how GetDatabase() routes StringIncrementAsync. Done means operations are not sent to A while it is rejoining as a replica.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, redis
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.