dapr / dapr/components-contrib
unstable work of rabbitmq pubsub with resilience policies (with the loss of messages!)
- Dominant language
- Go
- Stars
- 602
- Forks
- 580
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
Hello everyone!
We have encountered an important problem when using dapr resilience policies with rabbitmq pubsub.
Previously, we used kafka, and if some message got into the topic that could not be processed and for which the dead letter topic was not enabled, dapr continued trying to process this message every 5 seconds, and it was OK.
After migrating to rabbitmq, we encountered the problem that the broken message tried to be re-processed immediately after the crash, that is, it made thousands of attempts per minute.
To solve this problem, we added a resiliency policy for DefaultRetryPolicy, setting the frequency of retries every 5 seconds:
```
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
name: donum-resiliency
spec:
policies:
retries:
# Global Retry Policy
DefaultRetryPolicy:
policy: constant
duration: 5s
targets:
apps:
# apps and their applied policies here
```
It worked, retries began to occur every 5 seconds. We had one corrupted message left in the queue (a message about the need to send an SMS message, but with an incorrect number), and it turned out that the frequency of re-processing was increasing every hour, and after 18 hours the frequency reached 400-430 attempts per MINUTE instead of the expected 12 attempts. In fact, we made a DDoS attack on our SMS gateway.

We tried to re-deploy our applications and monitor the frequency of exceptions, and that bug was detected! I wasn't quite right. The frequency remains every 5 seconds, but at some point the message starts to be processed twice (but we have only one message).

Then we faced an even more important problem, which in my opinion is critical.
We assumed that the bug could only be in the constant policy, and tried to rewrite it to the exponential type:
```
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
name: donum-resiliency
spec:
policies:
retries:
# Global Retry Policy
DefaultPubsubComponentInboundRetryPolicy:
policy: exponential
duration: 5s
maxInterval: 60s
maxRetries: -1 # Retry indefinitely
targets:
apps:
# apps and their applied policies here
```
Firstly, we were faced with the fact that the retry period was unstable and went beyond the allowed values.
1) At first it was less than 5 seconds:

2) Then the period began to increase, but chaotically. The period could decrease by 3 times, and then increase immediately by 6 times, and even became more than 60 seconds, i.e. exceeded the specified maxInterval:


3) But the worst thing is that at some point message processing attempts ended, however, not because the message was successfully processed (it was impossible, it was broken), but because this message disappeared from the queue! It looks like dapr sent a successful ack to rabbitmq, although MaxRetries=-1 and a different behavior is specified in pubsub:
```
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: donum-pubsub
spec:
type: pubsub.rabbitmq
version: v1
metadata:
- name: host
value: "amqp://guest:guest@rabbitmq:5672"
- name: durable
value: true
- name: deletedWhenUnused
value: false
**- name: autoAck
value: false**
- name: deliveryMode
value: 2
**- name: requeueInFailure
value: true**
- name: prefetchCount
value: 0
- name: reconnectWait
value: 0
- name: concurrencyMode
value: parallel
- name: publisherConfirm
value: true
- name: enableDeadLetter
value: false
```
There is nothing new in the dapr sidecar logs, the last entry was still about the inability to process the message:

Contributor guide
Research direction
Start with the RabbitMQ pubsub component's failed-delivery handling and the resilience retry policy, using the supplied Resiliency and Component YAML as reproduction inputs. Compare retry timing, duplicate delivery, and final acknowledgement or requeue behavior against the stated maxRetries=-1 and requeueInFailure=true expectations. Done means the message is not lost or acknowledged before successful processing, and the reported retry behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, rabbitmq
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100