dapr / dapr/components-contrib
Azure EventHub PubSub component drastically reduces event distribution after a while
- Dominant language
- Go
- Stars
- 602
- Forks
- 580
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
## Expected Behavior
Unhandled events in the Azure EventHub get distributed to the Pods of the ReplicaSet until all events are handled.
## Actual Behavior
After (re-)starting the deployment, it runs as expected for a while. All pods receive events and run without issues.
After an hour or two, the first pod starts to only sporadically receive events. With more time, more and more pods display this behaviour until all of them only handle an event every few minutes or so.
The strange thing about it is that the amount of unhandled events is still way up, so it's not just that there are no events to handle.
Other notable details:
- The containers in the pods are healthy. If you could tell me a way to do it, I could also check whether the daprd sidecar responds.
- There are still unhandled events in the hub. I checked by looking at data that should've been changed by the pods, had the events been handled.
- The daprd sidecar shows no errors (debug logging is active).
## Steps to Reproduce the Problem
This is also a bit spicy. I have the exact same setup running in another namespace connected to another EventHub, running under different names and everything. No issues on this one.
Still, my setup includes:
- A Deployment listening via its sidecar to one dapr PubSub binding and publishing to another. It has the following annotations:
- dapr.io/app-protocol: grpc
- dapr.io/enable-api-logging: "true"
- dapr.io/enabled: "true"
- dapr.io/log-level: debug
- dapr.io/app-id: cloud-dev-sidedetection-logic
- dapr.io/app-port: "3010"
- A dapr PubSub component for subscription:
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: dev-sidedetection-sub
spec:
type: pubsub.azure.eventhubs
version: v1
metadata:
- name: connectionString
secretKeyRef:
key: AzureEventHubConnectionStringSubscriber
name: cloud-dev-sidedetection-logic
- name: storageAccountName
value: rwimportermaterialized
- name: storageAccountKey
secretKeyRef:
key: AzureEventCheckpointStorageKey
name: cloud-dev-sidedetection-logic
- name: storageContainerName
value: azure-webjobs-eventhub
- A dapr PubSub component for publishing:
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: dev-sidedetection-pub
spec:
type: pubsub.azure.eventhubs
version: v1
metadata:
- name: connectionString
secretKeyRef:
key: AzureEventHubConnectionStringPublisher
name: cloud-dev-sidedetection-logic
- name: storageAccountName
value: rwimportermaterialized
- name: storageAccountKey
secretKeyRef:
key: AzureEventCheckpointStorageKey
name: cloud-dev-sidedetection-logic
- name: storageContainerName
value: azure-webjobs-eventhub
- A Secret containing the connection strings
- A KEDA ScaledObject Custom Resource for scaling the ReplicaSet according to the amount of unhandled events
- A KEDA TriggerAuthentication CustomResource which just connects the ScaledObject to the secret
- A ClusterRoleBinding for HashicorpVault
I'd really like to give more information, but I'm highly unsure where to even look for more clues.
## Release Note
RELEASE NOTE:
Contributor guide
Research direction
Start by comparing the affected deployment, Dapr Azure EventHubs pubsub components, KEDA resources, and daprd debug logs with the otherwise identical healthy namespace. Confirm the reproduction over time and verify that all pods continue receiving events while the EventHub backlog is nonzero.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go
- Domain
- backend, cloud, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100