StackExchange / StackExchange/StackExchange.Redis

Unable to reconnect to the Redis server after RedisConnectionException.

Open
#2,829 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

💬 needs-info
Dominant language
C#
Stars
6.2k
Forks
1.6k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

Hi everyone, I have a net8 api project that runs locally. Through docker deployment, Redis Serve is also deployed in docker on the same host.

StackExchange.Redis Version 2.8.22

The connection string in the code is as follows

"127.0.0.1:6379,DefaultDatabase=0,allowAdmin=true,abortConnect=false,connectRetry=5,connectTimeout=3000,syncTimeout=3000,asyncTimeout=3000,name=MesBus_Api,keepAlive=30"

Under normal circumstances, I will see two client connections named MesBus_Api in Redis Serve.looks like this

id=48 addr=127.0.0.1:35494 laddr=127.0.0.1:6379 fd=14 name=MesBus_Api age=8 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=60 obl=0 oll=0 omem=0 tot-mem=22272 events=r cmd=setex user=default redir=-1 resp=2
id=49 addr=127.0.0.1:35496 laddr=127.0.0.1:6379 fd=15 name=MesBus_Api age=8 idle=7 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1856 events=r cmd=subscribe user=default redir=-1 resp=2

But from time to time I encounter the following error:

StackExchange.Redis.RedisConnectionException: No connection is active/available to service this operation: HMGET MesBus:TokenBLK:2b290cab2815ab802a7e18e01bf679e1, inst: 0, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, last-in: 2, cur-in: 0, sync-ops: 72, async-ops: 227416, serverEndpoint: 127.0.0.1:6379, conn-sec: 4370.5, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: hc-phytiumft20004(SE.Redis-v2.7.10.12442), IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=3,Free=32764,Min=16,Max=32767), POOL: (Threads=12,QueuedItems=0,CompletedItems=1155132,Timers=11), v: 2.7.10.12442
at StackExchange.Redis.ConnectionMultiplexer.ThrowFailed[T](TaskCompletionSource`1 source, Exception unthrownException) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2037

When I got this exception, I found that my project could not reconnect to the redis server correctly, and I used redis cli to check that the server had no slowlog, and the server was working normally (I have two programs written in rust that also use the redis server, both using db0)

Moreover, I checked the client list and there are still two MesBus_Api connections. When I restart the redis server, I can see that there is a MesBus_Api connection and my API project still tries to initiate a connection to redis because I detect that there is a TCP connection in my server to the Redis server, although They all end up as Time_Wait and they are unable to establish a normal connection to redis. Unless I restart my API project, at this point I can see that two connections named MesBus_Api have been established in redis. And the exception disappears in the project and all code works normally.

How should I troubleshoot or check this issue? Thank you for your help.
ps.I checked the server's CPU usage and it wasn't too high.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ConnectionMultiplexer.cs:2037 and compare the reported connection state with the Redis CLI client list before and after restarting Redis. Verify why the deployed application reports StackExchange.Redis v2.7.10 despite the stated 2.8.22 dependency, and use the provided Docker setup and connection string to reproduce the reconnect failure. Done means identifying the cause of the unusable connections and documenting a reliable recovery path.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, redis
Domain
database, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.