apache / apache/dubbo

Zookeeper集群短暂宕机后,集群恢复受阻

Open
#12,189 5 comments 0 reactions 0 assignees View on GitHub
type/discussion
Dominant language
Java
Stars
41.6k
Forks
26.4k
Avg merge
15h 13m
Merged PRs (30d)
4

Description

### Environment

* Dubbo version: 2.7.x
* Operating System version: Linux RedHat
* Java version: 1.8

### Steps to reproduce this issue

1. Zookeeper集群宕机过半节点,集群不可用
2. 在Zookeeper每个节点启动前,客户端依旧保持不断重试连接的状态
3. Zookeeper宕机节点启动缓慢,选主十分困难
4. 由于我们的设计是将Zookeeper设计成一级注册中心,注册服务较多。怀疑是客户端连接过多,瞬间过来的重连流量导致了网络阻塞,目前稳定下来后,2181端口大约有1.7k的tcp监听。

现有的一些ZK的数据:
1、实际服务大约1.8w,ZNode节点约33w
2、监听watcher数900多w

目前想到的方案只有扩容Zookeeper集群,毋庸置疑的,现在现象可以看到压力已经达到瓶颈,集群扩容可以减少瞬时重连流量的击垮风险。想来请教,大家实施过程中有没有遇到这个问题,是否还有其他可行方案?
是否还可以进行优化,客户端重连时添加随机时间,减少瞬时重连流量的概率?

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are named. Start by reproducing the ZooKeeper majority-outage scenario with Dubbo 2.7.x on Linux and Java 8, then investigate client reconnect behavior against the reported service, znode, watcher, and connection counts; done would require a confirmed mitigation and clearly defined recovery behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.