apache / apache/pulsar

slow consumer will cause backlog quota exceeded

Open
#15,284 5 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale Stale type/feature
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Is your feature request related to a problem? Please describe.**

if we have two consumers in one subscription, one is slow(connection is ok, but only slow), the it will hold some unacked messages. use `produce exception` strategy

after a while, backlogsize will increase, and new producers will get

Could not get connection to broker: Cannot create producer on topic with
backlog quota exceeded -- Will try again xx

**Describe the solution you'd like**

maybe we can add

```
public void handleExceededBacklogQuota(PersistentTopic persistentTopic) {
TopicName topicName = TopicName.get(persistentTopic.getName());
BacklogQuota quota = getBacklogQuota(topicName);
log.info("Backlog quota exceeded for topic [{}]. Applying [{}] policy", persistentTopic.getName(),
quota.getPolicy());
switch (quota.getPolicy()) {
case consumer_backlog_eviction:
dropBacklog(persistentTopic, quota);
break;
case producer_exception:
case producer_request_hold:
disconnectProducers(persistentTopic);
break;
case resend unacked: // add new strategy

default:
break;
}
}
```

or. in the dispatcher, record the timeout.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the PersistentTopic backlog-quota handling and the dispatcher behavior mentioned in the issue. Compare the proposed handleExceededBacklogQuota flow with recording a timeout for unacked messages, then define how the new resend-unacked strategy should interact with existing quota policies. Done means the behavior for a slow consumer and new producers is specified and verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.