StackExchange / StackExchange/StackExchange.Redis

StringIncrement() returning the same value on two different machines

Open
#1,886 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.2k
Forks
1.6k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

We are using StackExchange.Redis (v2.0.601) to connect to Azure Redis. We are using StringIncrement() to increment the value of a key while inserting a record into another storage system, and then stop inserting when the Redis value exceeds a certain number. This is running on multiple machines with each service multi-threaded. This is a high load scenario where we are hitting StringIncrement() with hundreds or even thousands of requests per second, spread across multiple machines.

What I am seeing is that we are actually inserting more records than we expect. When I log the values being returned by StringIncrement(), ** I see that we are getting the same value returned on different machines**. Here is the code we are executing:

returnedValue = AzureCache(clusterIndex).StringIncrement(cacheKey, quantity)

The list below is shows a count of the value returned and the count of them.
|returned-value-after-increment|count-of-this-value-across-all-machines|
So, the value 208 was returned to 4 different threads. The value 3 was returned to 3 different threads.

  | 208 | 4 |  
  | 207 | 4 |  
  | 211 | 4 |  
  | 209 | 4 |  
  | 206 | 4 |  
  | 210 | 4 |  
  | 205 | 4 |  
  | 213 | 4 |  
  | 212 | 4 |  
  | 3 | 3 |  
  | 4 | 3 |  
  | 215 | 3 |  
  | 2 | 3

I thought that StringIncrement() is atomic and could never return the same value to two different threads/machines. Can anyone explain what is going on and how to fix this issue? Am I misunderstanding how this should work?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named. Start from the StringIncrement(cacheKey, quantity) call against Azure Redis and reproduce the high-load, multi-machine scenario described with StackExchange.Redis v2.0.601; done means explaining the repeated returned values and identifying a verified fix or limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, redis
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.