apache / apache/pulsar

kill the broker or ack takes a long time, which will cause the cluster traffic to drop instantaneously

Open
#13,523 10 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale Stale type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Is your enhancement request related to a problem? Please describe.**
I have nine brokers,when I kill one, the traffic of the remaining eight brokers will drop within 30 seconds, reach a peak after 30 seconds, and then return to normal. I found that the sending thread is blocked:
```
private boolean canEnqueueRequest(SendCallback callback, long sequenceId, int payloadSize) {
try {
if (conf.isBlockIfQueueFull()) {
if (semaphore.isPresent()) {
semaphore.get().acquire();
}
..
```
In other words, if the Producer queue corresponding to some partitions is full, the sending thread will be blocked, affecting the sending of other partitions

**Describe the solution you'd like**
Isolate partitions to avoid mutual influence when sending exceptions

**Describe alternatives you've considered**
PartitionedProducerImpl can automatically exclude Producers whose queues are full

Contributor guide

Open the contributing guide

Research direction

Start with the canEnqueueRequest method and PartitionedProducerImpl, focusing on how semaphore acquisition and full producer queues affect sending across partitions. Reproduce the traffic drop when one broker is killed, then define and test behavior where a full or failing partition producer does not block other partitions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.