Redis Sentinel / Connection Leak

Redis Sentinel / Connection Leak

StackExchange.Redis (2.8.24) / OpenTelemetry.Instrumentation.StackExchangeRedis (1.9.0-beta.1)

Description

We are observing a connection leak in our production servers running redis sentinel.

First of all, we checked that there is no ConnectionMultiplexer instance leak in our code base and in the dump.

It turns out that when switching primary, the ConnectionMultiplexer's servers are cleared but the _serverSnapshot is not which causes duplicates (see below 1. figure) in the _serverSnapshot which pins PhysicalBridges (see below 2. and 3. figures).

1. Duplicates in _serverSnapshot
   

2. Leak of PhysicalBridge : 338
   

3. Similar Retention (max 204) with a common path:
   

It seems that when adding a new endpoint to the _serverSnapshot the servers field has been previously cleared and duplicates appear in the _serverSnapshot field.

How to reproduce

1. Setup a redis sentinel
2. Create a ConnectionMultiplexer with StackExchange.Redis (2.8.24)
3. Instrument with OpenTelemetry.Instrumentation.StackExchangeRedis (1.9.0-beta.1)
4. Switch primary / replica over and over
5. Observe the number of connections.

Start with ConnectionMultiplexer.Sentinel.cs around line 415 and ConnectionMultiplexer.cs around line 905, then reproduce repeated Sentinel primary/replica switches using the listed StackExchange.Redis and OpenTelemetry versions. Trace how clearing servers and adding endpoints updates _serverSnapshot and retains PhysicalBridges. Done means repeated failovers no longer create duplicate snapshot entries or growing retained connections.