Azure Queue stream provider does not delete messages if previous message has failed
- Dominant language
- C#
- Stars
- 10.9k
- Forks
- 2.1k
- Avg merge
- 14h 42m
- Merged PRs (30d)
- 354
Description
I believe I've found a bug in the Azure Queue Stream Provider package.
## Scenario
* Producer pushes message to stream, consumer encounters exception while processing the message.
* Message will be retried indefinitely if it continues to throw an exception in the consumer.
* Messages pushed to the same queue will also be retried indefinitely, even if they are consumed successfully, as long as the first failed message continues to fail in the consumer. That is, messages added after the failing message are not deleted from the queue and continue to be retried.
* Manually deleting the failing message allows the remaining messages to be deleted after consumer successfully processes the message.
## Setup
* Orleans running on AKS cluster
* Azure Storage Queue Stream Provider
* Azure blob storage for PubSub
## Steps to reproduce
1. Generate a stream message which will result in an exception by the consumer.
2. Generate stream messages on the same queue as the failing message.
3. Monitor dequeue count of messages, which increases indefinitely while failing message remains in queue. The failing message and good messages will all retry indefinitely.
4. Manually delete the failing message from the queue.
5. Monitor remaining messages on queue, which will process and be removed from queue by pulling agent.
Example:
Messages A and B
A throws exception in consumer and will be retried later. B is added to the same queue as A. The consumer processes B, but pulling agent does not remove B from the queue. A and B will remain in the queue indefinitely until A is manually deleted.
I was able to reproduce this in my environment.
## Summary
I was able to determine the source of the exceptions for my failed messages. I can remediate through validating dependencies prior to pushing to stream, but I think there is bug in the pulling agent. Messages which are consumed without exceptions should be removed from the queue even if another previous message continues to fail in the consumer.
Contributor guide
Assessment
This issue has not been assessed yet.