apache / apache/pulsar

[Bug] Broker failed to recover after restarting when consumers are not disconnected fully

Open
#17,190 1 comment 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

2.7

### Minimal reproduce step

Launch many 10s of thousands of clients, produce millions of messages per minute into thousands of partitioned topics.

Now kill zookeepers and bring them back. Restart brokers one by one.

You may see that brokers can die and being killed by themselves over and iver again, never recover

### What did you expect to see?

Clients receive 429 too many requests, however over some time brokers should all recover and serve normally.

### What did you see instead?

Cluster never recovers.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are named. Start by reproducing the Version 2.7 scenario with partitioned topics, many consumers, ZooKeeper restarts, and sequential broker restarts. Done means brokers recover and resume serving normally instead of repeatedly dying while clients receive temporary 429 responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.