dapr / dapr/components-contrib
[PubSub Kafka] : Subscriber stopped consuming and didn't auto-heal causing lag
- Dominant language
- Go
- Stars
- 602
- Forks
- 580
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 6
Description
**Dapr 1.17.3**
## Expected Behavior
When dapr looses it's subscription to Kafka, the dapr process auto-heals and continues consuming messages
## Actual Behavior
Dapr generated the following error, and then stopped processing any messages on that Topic, causing lag to grow.
`Recovered processing operation component[general-purpose-pubsub] input after 2 attempts`
Followed by this log, which was the last log event that dapr generated before going silent.
`Error processing Kafka message: ds-REDACTED-created/3/452 [key=]. Error: subscription is closed.`
I evicted the pod and the new pod was scheduled and all messages were instantly processed successfully. So dapr had lost its subscription and failed to auto-heal etc
## Steps to Reproduce the Problem
I've not been able to repro this despite trying.
I can provide full dapr container logs privately if they are required for investigative purposes.
However I can provide the following key details
- App Health checks are **enabled**
- Kafka PubSub has the metadata setting `consumeRetryEnabled` set to `false`
- Resiliency is applied to PubSub target, as follows :
```
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
name: dbx-retry-for-general-purpose-pubsub
spec:
policies:
retries:
pubsubRetry:
policy: constant
duration: 40s
maxRetries: 5
targets:
components:
general-purpose-pubsub:
inbound:
retry: pubsubRetry
scopes:
- dbx
```
## Release Note
RELEASE NOTE:
Contributor guide
Research direction
No file or test is named. Start by tracing the Kafka PubSub subscriber path around the “subscription is closed” error and how recovery interacts with consumeRetryEnabled and the configured resiliency policy. Done means a lost Kafka subscription recovers without pod eviction, message consumption resumes, and lag does not continue growing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100