StackExchange / StackExchange/StackExchange.Redis
RedisCommandException during Azure Cluster failover
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.2k
- Forks
- 1.6k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 43
Description
Configuration:
- StackExchange.Redis 2.5.61
- Azure Redis with clustering enabled, 2 shards (Redis version 6.0.14)
I've noticed RedisCommandException being thrown in WriteMessageToServerInsideWriteLock during the scheduled maintenance window.
"id": "3923059",
"outerId": "17459436",
"type": "StackExchange.Redis.RedisCommandException",
"message": "Command cannot be issued to a replica: HDEL <KEY>",
"parsedStack": [{
"level": 0,
"method": "StackExchange.Redis.PhysicalBridge.WriteMessageToServerInsideWriteLock",
"assembly": "StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46",
"fileName": "/_/src/StackExchange.Redis/PhysicalBridge.cs",
"line": 1411
}]
This issue seems to be related to #1504 which should be resolved by #1876.
It looks like it's caused by the planned failover. The topology change was not recognized by the library at the time the command was sent. We didn't have any retry policy set up for that particular exception so it caused a brief downtime of a part of the system (less than 30s).
The question here is: should we still expect this exception when rebooting one of the shards in a cluster? Maybe it's worth adding a guideline on how to handle that to the docs.
I've already updated SE.Redis to the latest version and put a retry policy for that in place but wanted to report anyway.
Thanks
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting src/StackExchange.Redis/PhysicalBridge.cs around line 1411 and compare the behavior with issues #1504 and #1876. Reproduce or verify the command-routing behavior during an Azure Redis cluster failover using the reported configuration. Done means the expected exception behavior is established and any required retry or documentation guidance is identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp, redis
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100