Data was added repeatedly to the queue unexpectedly without any warning
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 14h 42m
- Merged PRs (30d)
- 354
Description
### Describe the bug
When using [`AddQueueMessage`](https://github.com/dotnet/orleans/blob/e16ae892abb01515d781d0b30f1fa25379221fab/src/Azure/Orleans.Streaming.AzureStorage/Storage/AzureQueueDataManager.cs#L195-L212) method of `AzureQueueDataManager` to append a message to the queue (i.e., Azure Queue service), sometimes it will add the message repeatedly beyond expectation, and there is no warning of this. The reason is that `AddQueueMessage` uses [`SendMessageAsync`](https://github.com/dotnet/orleans/blob/e16ae892abb01515d781d0b30f1fa25379221fab/src/Azure/Orleans.Streaming.AzureStorage/Storage/AzureQueueDataManager.cs#L202) API to send the HTTP request to the remote persistent queue service. When transient network errors (e.g., client-side timeout) happen, the SDK retry mechanism for the API will send another request to add the same message again. Since both requests have succeeded, there will be redundant data in the queue, and the context of the queue will be changed unexpectedly. Furthermore, the application is totally unaware of this, so if some tasks rely on the queue context later, the results may become unexpected when retrieving the data from the queue.
For instance, suppose we want to append three different messages "A, B, C" to the queue in sequence. When appending "A", the transient errors can happen and append the "A" twice, so the queue will be "A, A, B, C". When popping up two elements from the queue, what we get will be "A, A" instead of "A, B", which is unanticipated.
### To Reproduce
We found some existing tests for Azure may help to reproduce this issue. For instance, test [AQ_Standalone_1](https://github.com/dotnet/orleans/blob/5ec98fb485ea89be192947878216a546f02e35d4/test/Extensions/TesterAzureUtils/AzureQueueDataManagerTests.cs#L52-L55) and [AQ_Standalone_2](https://github.com/dotnet/orleans/blob/5ec98fb485ea89be192947878216a546f02e35d4/test/Extensions/TesterAzureUtils/AzureQueueDataManagerTests.cs#L90-L95) could fail due to this bug by comparing the number of expected queue message with the real number got from `manager.GetApproximateMessageCount()`.
```
...
var inMessage = "Hello, World";
await manager.AddQueueMessage(inMessage); // which calls SendMessageAsync SDK API inside
Assert.Equal(1, await manager.GetApproximateMessageCount()); // Assertion fails because the real number returned from GetApproximateMessageCount() is two
...
```
### Expected behavior and fix
The message should not be appended to the queue repeatedly without the client's awareness. One possible way to fix this in Orleans could be to check the number of the queue message before and after adding a new one via `GetApproximateMessageCount`. If the number exceeds what we expect, we should remove (pop up) the redundant ones.
### Additional context
This bug is similar to #7738
Contributor guide
Assessment
This issue has not been assessed yet.