StackExchange / StackExchange/StackExchange.Redis

No logs or events for issues with password auth on enterprise instances when using multiple connection strings

Open
#2,869 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.2k
Forks
1.6k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

We recently ran into an issue in our environment where we overloaded the TLS negotiation of our enterprise redis instances, and the reason for it was because we were passing multiple connection strings to the multiplexer that used different passwords (which we now understand is not supported). This resulted in the multiplexer spamming the redis instances with connection requests because it kept running into auth failure, and we could see on the redis logs when we profile them a large repeated spam of these commands per second, per app:

15:09:11.938 "AUTH" "(redacted)"
15:09:11.938 "CLIENT" "SETNAME" "AppName(SE.Redis-v2.8.31.52602)"
15:09:11.938 "CLIENT" "SETINFO" "lib-name" "SE.Redis"
15:09:11.938 "CLIENT" "SETINFO" "lib-ver" "2.8.31.52602"
15:09:11.938 "CLIENT" "ID"
15:09:11.938 "CONFIG" "GET" "replica-read-only"
15:09:11.938 "CONFIG" "GET" "databases"
15:09:11.938 "SENTINEL" "MASTERS"
15:09:11.938 "INFO" "replication"
15:09:11.938 "INFO" "server"
15:09:11.938 "CLUSTER" "NODES"
15:09:11.938 "GET" "__Booksleeve_TieBreak"
15:09:11.938 "ECHO" "n \xc9jj'SB\xb4\x0b\xe6E\xf1\xd7\xec["

We had a very difficult time triaging this issue because we couldn't find any logs or errors anywhere. We tried subscribing to the events the multiplexer offers, like so:

var multiplexer = ConnectionMultiplexer.Connect($"{redisConnectionString},{redisSecondaryConnectionString},connectRetry=3");
multiplexer.ConnectionFailed += this.Redis_ConnectionFailed;
multiplexer.ConnectionRestored += this.Redis_ConnectionRestored;
multiplexer.ErrorMessage += this.Redis_ErrorMessage;
multiplexer.InternalError += this.Redis_InternalError;

But none of them were firing during these issues so we had nothing to log. There were no exceptions being thrown/logged from the library either. How would we go about detecting when the library runs into an auth issue and fails to properly connect? From what the redis team was telling us, it was stuck in a loop of Connection Open > send auth command > fail auth > Connection Close, repeat.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the connection loop with multiple enterprise Redis connection strings using different passwords, then inspect which connection and error events are emitted during authentication failure. Done means authentication failures are surfaced through a usable log, event, or exception path so applications can detect the failed connections.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, redis
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.