StackExchange / StackExchange/StackExchange.Redis
Unbalanced shard set reads (1.2.6 -> 2.6.80)
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.2k
- Forks
- 1.6k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 43
Description
Since upgrading our project from StackExchange.Redis version 1.2.6 to 2.6.80, we have noticed that reads are unbalanced between the 2 nodes in the shard set.
We are using Azure Cache for Redis with 2 shards in the cluster.
Prior to the update there was approximately 10%-20% difference between reads across the shards, however the difference is now 6-7X with one shard taking the majority of the load.
Most metrics are balanced across the 2 shards: connections, cache hits & misses, gets & sets, used memory, total keys.
Metrics including: Total operations, CPU, Server Load, and Reads, all exhibit significant difference across the 2 shard nodes.
We do make heavy use of pub/sub with a single channel.
Upgrade occurred on Jan 30.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported upgrade from StackExchange.Redis 1.2.6 to 2.6.80 on Azure Cache for Redis with two shards. Investigate the single-channel pub/sub usage and compare the listed shard metrics across both versions; done means identifying a reproducible cause or confirming the behavior cannot be isolated from the provided information.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, redis
- Domain
- databases, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100