dapr / dapr/components-contrib

Azure EventHub PubSub component drastically reduces event distribution after a while

Open
#2,280 4 comments 0 reactions 0 assignees View on GitHub
kind/bug pinned
Dominant language
Go
Stars
602
Forks
580
Avg merge
4d 9h
Merged PRs (30d)
6

Description

## Expected Behavior

Unhandled events in the Azure EventHub get distributed to the Pods of the ReplicaSet until all events are handled.

## Actual Behavior

After (re-)starting the deployment, it runs as expected for a while. All pods receive events and run without issues.
After an hour or two, the first pod starts to only sporadically receive events. With more time, more and more pods display this behaviour until all of them only handle an event every few minutes or so.
The strange thing about it is that the amount of unhandled events is still way up, so it's not just that there are no events to handle.
Other notable details:
- The containers in the pods are healthy. If you could tell me a way to do it, I could also check whether the daprd sidecar responds.
- There are still unhandled events in the hub. I checked by looking at data that should've been changed by the pods, had the events been handled.
- The daprd sidecar shows no errors (debug logging is active).

## Steps to Reproduce the Problem

This is also a bit spicy. I have the exact same setup running in another namespace connected to another EventHub, running under different names and everything. No issues on this one.

Still, my setup includes:

- A Deployment listening via its sidecar to one dapr PubSub binding and publishing to another. It has the following annotations:
- dapr.io/app-protocol: grpc
- dapr.io/enable-api-logging: "true"
- dapr.io/enabled: "true"
- dapr.io/log-level: debug
- dapr.io/app-id: cloud-dev-sidedetection-logic
- dapr.io/app-port: "3010"
- A dapr PubSub component for subscription:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: dev-sidedetection-sub
spec:
type: pubsub.azure.eventhubs
version: v1
metadata:
- name: connectionString
secretKeyRef:
key: AzureEventHubConnectionStringSubscriber
name: cloud-dev-sidedetection-logic
- name: storageAccountName
value: rwimportermaterialized
- name: storageAccountKey
secretKeyRef:
key: AzureEventCheckpointStorageKey
name: cloud-dev-sidedetection-logic
- name: storageContainerName
value: azure-webjobs-eventhub
- A dapr PubSub component for publishing:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: dev-sidedetection-pub
spec:
type: pubsub.azure.eventhubs
version: v1
metadata:
- name: connectionString
secretKeyRef:
key: AzureEventHubConnectionStringPublisher
name: cloud-dev-sidedetection-logic
- name: storageAccountName
value: rwimportermaterialized
- name: storageAccountKey
secretKeyRef:
key: AzureEventCheckpointStorageKey
name: cloud-dev-sidedetection-logic
- name: storageContainerName
value: azure-webjobs-eventhub
- A Secret containing the connection strings
- A KEDA ScaledObject Custom Resource for scaling the ReplicaSet according to the amount of unhandled events
- A KEDA TriggerAuthentication CustomResource which just connects the ScaledObject to the secret
- A ClusterRoleBinding for HashicorpVault

I'd really like to give more information, but I'm highly unsure where to even look for more clues.

## Release Note

RELEASE NOTE:

Contributor guide

Open the contributing guide

Research direction

Start by comparing the affected deployment, Dapr Azure EventHubs pubsub components, KEDA resources, and daprd debug logs with the otherwise identical healthy namespace. Confirm the reproduction over time and verify that all pods continue receiving events while the EventHub backlog is nonzero.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go
Domain
backend, cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.