dapr / dapr/components-contrib

unstable work of rabbitmq pubsub with resilience policies (with the loss of messages!)

Open
#2,557 13 comments 0 reactions 0 assignees View on GitHub
help wanted kind/bug
Dominant language
Go
Stars
602
Forks
580
Avg merge
4d 9h
Merged PRs (30d)
6

Description

Hello everyone!

We have encountered an important problem when using dapr resilience policies with rabbitmq pubsub.

Previously, we used kafka, and if some message got into the topic that could not be processed and for which the dead letter topic was not enabled, dapr continued trying to process this message every 5 seconds, and it was OK.

After migrating to rabbitmq, we encountered the problem that the broken message tried to be re-processed immediately after the crash, that is, it made thousands of attempts per minute.

To solve this problem, we added a resiliency policy for DefaultRetryPolicy, setting the frequency of retries every 5 seconds:
```
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
name: donum-resiliency
spec:
policies:
retries:
# Global Retry Policy
DefaultRetryPolicy:
policy: constant
duration: 5s

targets:
apps:
# apps and their applied policies here
```

It worked, retries began to occur every 5 seconds. We had one corrupted message left in the queue (a message about the need to send an SMS message, but with an incorrect number), and it turned out that the frequency of re-processing was increasing every hour, and after 18 hours the frequency reached 400-430 attempts per MINUTE instead of the expected 12 attempts. In fact, we made a DDoS attack on our SMS gateway.
![image](https://user-images.githubusercontent.com/44698/219905595-d89724de-d4d6-4424-b9e4-9687657243a0.png)

We tried to re-deploy our applications and monitor the frequency of exceptions, and that bug was detected! I wasn't quite right. The frequency remains every 5 seconds, but at some point the message starts to be processed twice (but we have only one message).
![image](https://user-images.githubusercontent.com/44698/219905670-5a36b65b-74e5-4aba-9dc5-31458d98e93b.png)

Then we faced an even more important problem, which in my opinion is critical.
We assumed that the bug could only be in the constant policy, and tried to rewrite it to the exponential type:
```
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
name: donum-resiliency
spec:
policies:
retries:
# Global Retry Policy
DefaultPubsubComponentInboundRetryPolicy:
policy: exponential
duration: 5s
maxInterval: 60s
maxRetries: -1 # Retry indefinitely
targets:
apps:
# apps and their applied policies here
```

Firstly, we were faced with the fact that the retry period was unstable and went beyond the allowed values.
1) At first it was less than 5 seconds:
image

2) Then the period began to increase, but chaotically. The period could decrease by 3 times, and then increase immediately by 6 times, and even became more than 60 seconds, i.e. exceeded the specified maxInterval:
![image](https://user-images.githubusercontent.com/44698/219906369-358d309d-04c3-44d1-8d3b-24a1cd8345c5.png)

![image](https://user-images.githubusercontent.com/44698/219906456-ea548272-9a62-4341-abaa-b0ae9ffec471.png)

3) But the worst thing is that at some point message processing attempts ended, however, not because the message was successfully processed (it was impossible, it was broken), but because this message disappeared from the queue! It looks like dapr sent a successful ack to rabbitmq, although MaxRetries=-1 and a different behavior is specified in pubsub:
```
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: donum-pubsub
spec:
type: pubsub.rabbitmq
version: v1
metadata:
- name: host
value: "amqp://guest:guest@rabbitmq:5672"
- name: durable
value: true
- name: deletedWhenUnused
value: false
**- name: autoAck
value: false**
- name: deliveryMode
value: 2
**- name: requeueInFailure
value: true**
- name: prefetchCount
value: 0
- name: reconnectWait
value: 0
- name: concurrencyMode
value: parallel
- name: publisherConfirm
value: true
- name: enableDeadLetter
value: false
```

There is nothing new in the dapr sidecar logs, the last entry was still about the inability to process the message:
image

Contributor guide

Open the contributing guide

Research direction

Start with the RabbitMQ pubsub component's failed-delivery handling and the resilience retry policy, using the supplied Resiliency and Component YAML as reproduction inputs. Compare retry timing, duplicate delivery, and final acknowledgement or requeue behavior against the stated maxRetries=-1 and requeueInFailure=true expectations. Done means the message is not lost or acknowledged before successful processing, and the reported retry behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, rabbitmq
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.