dapr / dapr/components-contrib
Dapr kafka endpoint for azure eventhub receiving duplicated messages
- Dominant language
- Go
- Stars
- 602
- Forks
- 580
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
I am using Dapr kafka endpoint for azure eventhub. I have created 3 partitions in the eventhub. I have 3 pods in kubernetes running to read events from eventhub. When I publish 3 events, each event is received by all the 3 consumers.
Every message is processed by all the three consumers. It is resulting in wastage of resources and duplication of outputs. Sometimes, the event is consumed twice or even thrice by the same consumer.
Each events take approx. 4 minutes to process.
All these three consumers belong to the same consumer group.
Below is my yaml file:
```
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: pubsub
spec:
type: pubsub.kafka
version: v1
metadata:
- name: brokers
value: "my_evenhtub_namespace_name.servicebus.windows.net:9093"
- name: consumerGroup
value: "consumer1"
- name: authRequired
value: "true"
- name: authType
value: "password"
- name: saslUsername
value: $ConnectionString
- name: saslPassword
secretKeyRef:
name: eventhub-connection-key
key: eventhub-connection-key
- name: version # Optional.
value: 1.0.0
- name: initialOffset
value: "oldest"
- name: consumeRetryInterval
value: 6s
auth:
secretStore: secretref
```
```
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
name: consumer-resiliency
spec:
policies:
retries:
pubsubRetry:
policy: constant
duration: 10s
maxRetries: 1
targets:
components:
pubsub:
inbound:
retry: pubsubRetry
```
```
apiVersion: dapr.io/v1alpha1
kind: Subscription
metadata:
name: consumer
spec:
topic: dev
route: /process
pubsubname: pubsub
deadLetterTopic: poison-message
scopes:
- consumer-dev
---
apiVersion: dapr.io/v1alpha1
kind: Subscription
metadata:
name: recorder-consumer-deadletter
spec:
topic: dead-message
route: /dead
pubsubname: pubsub
scopes:
- consumer-dev
```
Contributor guide
Research direction
Start with the Kafka component YAML, especially consumerGroup, consumeRetryInterval, initialOffset, and the resiliency retry policy, then reproduce with three Kubernetes pods, three Event Hubs partitions, and three published events. Check delivery to the /process route while accounting for the four-minute processing time; done means identifying why messages reach multiple consumers or are retried and documenting a verified fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, kafka
- Domain
- cloud, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100