dapr / dapr/components-contrib

[PubSub Kafka] : Subscriber stopped consuming and didn't auto-heal causing lag

Open
#4,349 4 comments 0 reactions 0 assignees View on GitHub
kind/bug pinned
Dominant language
Go
Stars
602
Forks
580
Avg merge
4d 9h
Merged PRs (30d)
6

Description

**Dapr 1.17.3**

## Expected Behavior

When dapr looses it's subscription to Kafka, the dapr process auto-heals and continues consuming messages

## Actual Behavior

Dapr generated the following error, and then stopped processing any messages on that Topic, causing lag to grow.

`Recovered processing operation component[general-purpose-pubsub] input after 2 attempts`

Followed by this log, which was the last log event that dapr generated before going silent.

`Error processing Kafka message: ds-REDACTED-created/3/452 [key=]. Error: subscription is closed.`

I evicted the pod and the new pod was scheduled and all messages were instantly processed successfully. So dapr had lost its subscription and failed to auto-heal etc

## Steps to Reproduce the Problem

I've not been able to repro this despite trying.

I can provide full dapr container logs privately if they are required for investigative purposes.

However I can provide the following key details

- App Health checks are **enabled**
- Kafka PubSub has the metadata setting `consumeRetryEnabled` set to `false`
- Resiliency is applied to PubSub target, as follows :

```
apiVersion: dapr.io/v1alpha1
kind: Resiliency
metadata:
name: dbx-retry-for-general-purpose-pubsub
spec:
policies:
retries:
pubsubRetry:
policy: constant
duration: 40s
maxRetries: 5
targets:
components:
general-purpose-pubsub:
inbound:
retry: pubsubRetry
scopes:
- dbx
```

## Release Note

RELEASE NOTE:

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by tracing the Kafka PubSub subscriber path around the “subscription is closed” error and how recovery interacts with consumeRetryEnabled and the configured resiliency policy. Done means a lost Kafka subscription recovers without pod eviction, message consumption resumes, and lag does not continue growing.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.