dotnet / dotnet/orleans

Redis Sentinel connection fails intermittently with "It was not possible to connect to the redis server(s)" error

Open
#9,754 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
10.9k
Forks
2.1k
Avg merge
14h 42m
Merged PRs (30d)
354

Description

Description
I'm using Orleans with Redis Sentinel as the persistence provider, and I'm experiencing intermittent connection failures during silo startup. The error occurs occasionally when the silo attempts to initialize the Redis storage.
Environment
Orleans Version: 9
Orleans.Persistence.Redis Version: 7
Redis Configuration: Redis Sentinel
Runtime: .NET 9
Deployment: EKS

Unhandled exception. Orleans.Persistence.Redis.RedisStorageException: StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code.
at Orleans.Persistence.RedisGrainStorage.Init(CancellationToken cancellationToken) in /_/src/Redis/Orleans.Persistence.Redis/Storage/RedisGrainStorage.cs:line 100
at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct) in /_/src/Orleans.Runtime/Lifecycle/SiloLifecycleSubject.cs:line 134
at Orleans.LifecycleSubject.OnStart(CancellationToken cancellationToken) in /_/src/Orleans.Core/Lifecycle/LifecycleSubject.cs:line 117
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute() in /_/src/Orleans.Runtime/Scheduler/ClosureWorkItem.cs:line 33
at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Silo/Silo.cs:line 180
at Orleans.Hosting.SiloHostedService.StartAsync(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Hosting/SiloHostedService.cs:line 28
at Microsoft.Extensions.Hosting.Internal.Host.b__14_1(IHostedService service, CancellationToken token)
at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
at Microsoft.Extensions.Hosting.Internal.Host.g__LogAndRethrow|14_3(<>c__DisplayClass14_0&)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Program.$(String[] args) in /src/src/BM.RoomManager.Host/Program.cs:line 364
at Program.(String[] args)

Contributor guide

Open the contributing guide

Research direction

Start at src/Redis/Orleans.Persistence.Redis/Storage/RedisGrainStorage.cs:100 and trace the Redis storage initialization path during silo startup. Reproduce the intermittent Redis Sentinel failure in the reported Orleans 9, Redis persistence, .NET 9, and EKS environment; the issue does not specify the expected retry or startup behavior, so the desired definition of done remains unclear.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, redis
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.