Zookeeper集群短暂宕机后,集群恢复受阻
- Dominant language
- Java
- Stars
- 41.6k
- Forks
- 26.4k
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 4
Description
### Environment
* Dubbo version: 2.7.x
* Operating System version: Linux RedHat
* Java version: 1.8
### Steps to reproduce this issue
1. Zookeeper集群宕机过半节点,集群不可用
2. 在Zookeeper每个节点启动前,客户端依旧保持不断重试连接的状态
3. Zookeeper宕机节点启动缓慢,选主十分困难
4. 由于我们的设计是将Zookeeper设计成一级注册中心,注册服务较多。怀疑是客户端连接过多,瞬间过来的重连流量导致了网络阻塞,目前稳定下来后,2181端口大约有1.7k的tcp监听。
现有的一些ZK的数据:
1、实际服务大约1.8w,ZNode节点约33w
2、监听watcher数900多w
目前想到的方案只有扩容Zookeeper集群,毋庸置疑的,现在现象可以看到压力已经达到瓶颈,集群扩容可以减少瞬时重连流量的击垮风险。想来请教,大家实施过程中有没有遇到这个问题,是否还有其他可行方案?
是否还可以进行优化,客户端重连时添加随机时间,减少瞬时重连流量的概率?
Contributor guide
Research direction
No source files, tests, or entry points are named. Start by reproducing the ZooKeeper majority-outage scenario with Dubbo 2.7.x on Linux and Java 8, then investigate client reconnect behavior against the reported service, znode, watcher, and connection counts; done would require a confirmed mitigation and clearly defined recovery behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100