StackExchange / StackExchange/StackExchange.Redis

Reading from streams with command flag "PreferReplica", does not recover after rebooting replica node

Open
#2,020 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

⚙️ area:cluster
Dominant language
C#
Stars
6.2k
Forks
1.6k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

We have an Azure Redis cluster (2 masters, 2 replicas), and are using SE version 2.2.88 in a .NET Core 3.1 service. We are reading streams from the replica nodes (using PreferReplica command flag). After we reboot the replica node where the stream is located, we get continuous errors when trying to read from the stream, it never seems to recover. The error that we get is Key has MOVED to Endpoint xx.xx.xx.xx:port and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD. We are able to consistently reproduce this issue with the code snippet below. We are assuming this issue is related to streams, because if we replace StreamReadAsync with StringGetAsync we can't reproduce the issue. Also, we can't reproduce the issue if we read the stream without specifying the command flag, and reboot the master node.

var options = ConfigurationOptions.Parse(
    "host123:1234,password=123,ssl=True,abortConnect=False");

options.CommandMap = CommandMap.Create(new HashSet<string> { "SUBSCRIBE" }, false);
options.CertificateValidation += (sender, certificate, chain, sslPolicyErrors) => true;

var connectionMultiplexer = await ConnectionMultiplexer.ConnectAsync(options);

const string key = "{06S}TestPreferReplicaStream";

await connectionMultiplexer.GetDatabase().StreamAddAsync(key, "Field", "hello world from shard 0");

await connectionMultiplexer.GetDatabase().KeyExpireAsync(key, TimeSpan.FromHours(1));

while (true)
{
    try
    {
        await connectionMultiplexer.GetDatabase().StreamReadAsync(key, "0-0",
            1000, CommandFlags.PreferReplica);
    }
    catch (Exception e)
    {
        await Task.Delay(100);
        Console.WriteLine("------------------------------------------------------------------------------------------------------------");
        Console.WriteLine($"{DateTime.Now}: " + e);
    }
}

Below is the output from the code snippet above.

------------------------------------------------------------------------------------------------------------
3/3/2022 1:39:18 PM: StackExchange.Redis.RedisConnectionException: SocketFailure on xx.xxx.xx.xx:15000/Interactive, Idle/Faulted, last: XREAD, origin: ReadFromPipe, outstanding: 1, last-read: 0s ago, last-write: 0s ago, unanswered-write: 0s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 9 of 10 available, in: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.2.88.56325
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at Pipelines.Sockets.Unofficial.StreamConnection.AsyncStreamPipe.CopyFromStreamToReadPipe() in /_/src/Pipelines.Sockets.Unofficial/StreamConnection.AsyncStreamPipe.cs:line 94
   at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
   at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
   at System.IO.Pipelines.Pipe.GetReadAsyncResult()
   at StackExchange.Redis.PhysicalConnection.ReadFromPipe()
   --- End of inner exception stack trace ---
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:39:40 PM: StackExchange.Redis.RedisConnectionException: UnableToConnect on xx.xxx.xx.xx:15000/Interactive, Initializing/NotStarted, last: XREAD, origin: BeginConnectAsync, outstanding: 0, last-read: 1s ago, last-write: 1s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, last-mbeat: 0s ago, global: 0s ago, v: 2.2.88.56325
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:39:42 PM: StackExchange.Redis.RedisConnectionException: UnableToConnect on xx.xxx.xx.xx:15000/Interactive, Initializing/NotStarted, last: XREAD, origin: BeginConnectAsync, outstanding: 0, last-read: 1s ago, last-write: 1s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, last-mbeat: 0s ago, global: 0s ago, v: 2.2.88.56325
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:29 PM: StackExchange.Redis.RedisConnectionException: UnableToConnect on xx.xxx.xx.xx:15000/Interactive, Initializing/NotStarted, last: XREAD, origin: BeginConnectAsync, outstanding: 0, last-read: 5s ago, last-write: 5s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, last-mbeat: 0s ago, global: 0s ago, v: 2.2.88.56325
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:54 PM: StackExchange.Redis.RedisTimeoutException: The timeout was reached before the message could be written to the output buffer, and it was not sent, command=XREAD, timeout: 5000, inst: 0, qu: 10, qs: 0, aw: True, bw: RecordingTimeout, rs: ReadAsync, ws: Flushed, in: 0, serverEndpoint: xx.xxx.xx.xx:15000, mc: 1/1/0, mgr: 10 of 10 available, clientName: S-TB-W9, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=0,Free=32767,Min=8,Max=32767), v: 2.2.88.56325 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:55 PM: StackExchange.Redis.RedisConnectionException: Loading on xx.xxx.xx.xx:15000/Interactive, Idle/ComputeResult, last: ECHO, origin: SetResult, outstanding: 0, last-read: 0s ago, last-write: 0s ago, keep-alive: 60s, state: ConnectedEstablishing, mgr: 9 of 10 available, last-heartbeat: never, last-mbeat: 0s ago, global: 0s ago, v: 2.2.88.56325
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:56 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:57 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:58 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:40:59 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:00 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:01 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:02 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:03 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:04 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:05 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:06 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:07 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: NOAUTH Authentication required.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:08 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
3/3/2022 1:41:09 PM: StackExchange.Redis.RedisServerException: Key has MOVED to Endpoint xx.xxx.xx.xx:15001 and hashslot 0 but CommandFlags.NoRedirect was specified - redirect not followed for XREAD.
   at TestAppCore.Program.Main(String[] args) in C:\VSProjects\TestAppCore\TestAppCore\Program.cs:line 37
------------------------------------------------------------------------------------------------------------
Keeps on going with same error

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided .NET Core reproduction using StreamReadAsync and CommandFlags.PreferReplica against the Azure Redis cluster. Trace how XREAD handles the MOVED response and NoRedirect after the replica reboot, comparing it with StringGetAsync and reads without PreferReplica. Done means stream reads recover after the replica returns without repeated MOVED, connection, or authentication errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, redis
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.