StackExchange / StackExchange/StackExchange.Redis

RedisCommandException during Azure Cluster failover

Open
#2,181 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.2k
Forks
1.6k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

Configuration:

  • StackExchange.Redis 2.5.61
  • Azure Redis with clustering enabled, 2 shards (Redis version 6.0.14)

I've noticed RedisCommandException being thrown in WriteMessageToServerInsideWriteLock during the scheduled maintenance window.

"id": "3923059",
"outerId": "17459436",
"type": "StackExchange.Redis.RedisCommandException",
"message": "Command cannot be issued to a replica: HDEL <KEY>",
"parsedStack": [{
  "level": 0,
  "method": "StackExchange.Redis.PhysicalBridge.WriteMessageToServerInsideWriteLock",
  "assembly": "StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46",
  "fileName": "/_/src/StackExchange.Redis/PhysicalBridge.cs",
  "line": 1411
}]

This issue seems to be related to #1504 which should be resolved by #1876.
It looks like it's caused by the planned failover. The topology change was not recognized by the library at the time the command was sent. We didn't have any retry policy set up for that particular exception so it caused a brief downtime of a part of the system (less than 30s).

The question here is: should we still expect this exception when rebooting one of the shards in a cluster? Maybe it's worth adding a guideline on how to handle that to the docs.

I've already updated SE.Redis to the latest version and put a retry policy for that in place but wanted to report anyway.
Thanks

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting src/StackExchange.Redis/PhysicalBridge.cs around line 1411 and compare the behavior with issues #1504 and #1876. Reproduce or verify the command-routing behavior during an Azure Redis cluster failover using the reported configuration. Done means the expected exception behavior is established and any required retry or documentation guidance is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, redis
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.