[Bug] Broker failed to recover after restarting when consumers are not disconnected fully
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
2.7
### Minimal reproduce step
Launch many 10s of thousands of clients, produce millions of messages per minute into thousands of partitioned topics.
Now kill zookeepers and bring them back. Restart brokers one by one.
You may see that brokers can die and being killed by themselves over and iver again, never recover
### What did you expect to see?
Clients receive 429 too many requests, however over some time brokers should all recover and serve normally.
### What did you see instead?
Cluster never recovers.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
No source files, tests, or entry points are named. Start by reproducing the Version 2.7 scenario with partitioned topics, many consumers, ZooKeeper restarts, and sequential broker restarts. Done means brokers recover and resume serving normally instead of repeatedly dying while clients receive temporary 429 responses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100