Redis Sentinel connection fails intermittently with "It was not possible to connect to the redis server(s)" error
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 14h 42m
- Merged PRs (30d)
- 354
Description
Description
I'm using Orleans with Redis Sentinel as the persistence provider, and I'm experiencing intermittent connection failures during silo startup. The error occurs occasionally when the silo attempts to initialize the Redis storage.
Environment
Orleans Version: 9
Orleans.Persistence.Redis Version: 7
Redis Configuration: Redis Sentinel
Runtime: .NET 9
Deployment: EKS
Unhandled exception. Orleans.Persistence.Redis.RedisStorageException: StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code.
at Orleans.Persistence.RedisGrainStorage.Init(CancellationToken cancellationToken) in /_/src/Redis/Orleans.Persistence.Redis/Storage/RedisGrainStorage.cs:line 100
at Orleans.Runtime.SiloLifecycleSubject.MonitoredObserver.OnStart(CancellationToken ct) in /_/src/Orleans.Runtime/Lifecycle/SiloLifecycleSubject.cs:line 134
at Orleans.LifecycleSubject.OnStart(CancellationToken cancellationToken) in /_/src/Orleans.Core/Lifecycle/LifecycleSubject.cs:line 117
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem.Execute() in /_/src/Orleans.Runtime/Scheduler/ClosureWorkItem.cs:line 33
at Orleans.Runtime.Silo.StartAsync(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Silo/Silo.cs:line 180
at Orleans.Hosting.SiloHostedService.StartAsync(CancellationToken cancellationToken) in /_/src/Orleans.Runtime/Hosting/SiloHostedService.cs:line 28
at Microsoft.Extensions.Hosting.Internal.Host.b__14_1(IHostedService service, CancellationToken token)
at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
at Microsoft.Extensions.Hosting.Internal.Host.g__LogAndRethrow|14_3(<>c__DisplayClass14_0&)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Program.$(String[] args) in /src/src/BM.RoomManager.Host/Program.cs:line 364
at Program.(String[] args)
Contributor guide
Research direction
Start at src/Redis/Orleans.Persistence.Redis/Storage/RedisGrainStorage.cs:100 and trace the Redis storage initialization path during silo startup. Reproduce the intermittent Redis Sentinel failure in the reported Orleans 9, Redis persistence, .NET 9, and EKS environment; the issue does not specify the expected retry or startup behavior, so the desired definition of done remains unclear.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, redis
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100