dapr / dapr/components-contrib

Dapr kafka endpoint for azure eventhub receiving duplicated messages

Open
#2,946 6 comments 0 reactions 0 assignees View on GitHub
pinned stale
Dominant language
Go
Stars
602
Forks
580
Avg merge
4d 9h
Merged PRs (30d)
6

Description

I am using Dapr kafka endpoint for azure eventhub. I have created 3 partitions in the eventhub. I have 3 pods in kubernetes running to read events from eventhub. When I publish 3 events, each event is received by all the 3 consumers.

Every message is processed by all the three consumers. It is resulting in wastage of resources and duplication of outputs. Sometimes, the event is consumed twice or even thrice by the same consumer.

Each events take approx. 4 minutes to process.

All these three consumers belong to the same consumer group.

Below is my yaml file:

```
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: pubsub
spec:
type: pubsub.kafka
version: v1
metadata:
- name: brokers
value: "my_evenhtub_namespace_name.servicebus.windows.net:9093"
- name: consumerGroup
value: "consumer1"
- name: authRequired
value: "true"
- name: authType
value: "password"
- name: saslUsername
value: $ConnectionString
- name: saslPassword
secretKeyRef:
name: eventhub-connection-key
key: eventhub-connection-key
- name: version # Optional.
value: 1.0.0
- name: initialOffset
value: "oldest"
- name: consumeRetryInterval
value: 6s
auth:
secretStore: secretref
```

```
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
name: consumer-resiliency
spec:
policies:
retries:
pubsubRetry:
policy: constant
duration: 10s
maxRetries: 1
targets:
components:
pubsub:
inbound:
retry: pubsubRetry
```

```
apiVersion: dapr.io/v1alpha1
kind: Subscription
metadata:
name: consumer
spec:
topic: dev
route: /process
pubsubname: pubsub
deadLetterTopic: poison-message
scopes:
- consumer-dev
---
apiVersion: dapr.io/v1alpha1
kind: Subscription
metadata:
name: recorder-consumer-deadletter
spec:
topic: dead-message
route: /dead
pubsubname: pubsub
scopes:
- consumer-dev
```

Contributor guide

Open the contributing guide

Research direction

Start with the Kafka component YAML, especially consumerGroup, consumeRetryInterval, initialOffset, and the resiliency retry policy, then reproduce with three Kubernetes pods, three Event Hubs partitions, and three published events. Check delivery to the /process route while accounting for the four-minute processing time; done means identifying why messages reach multiple consumers or are retried and documenting a verified fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, kafka
Domain
cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.