confluentinc / confluentinc/common
Zookeeper throws timeout after several restarts of nodes
- Dominant language
- Java
- Stars
- 17
- Forks
- 243
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 38
Description
I have a problem with zookeeper deployed on kubernetes using Confluentinc Docker Images.
### Cluster setup: ###
- 3 nodes in digital ocean ( 4 CPU, 8 GB or RAM) as kafka dedicated nodepool. ( Only zookeeper and kafka are running here, and some monitoring daemon set).
- If a zookeeper node/pod ( any node) restarts several times the zookeeper cluster is not accessible anymore, and to make it work I have to redeploy all the nodes.
- Once the node rejoined the cluster, all zookeeper nodes start to show this kind of error:
Each node has one kafka pod and one zookeeper pod.
### Errors one each node: ###
`zookeeper-node-0 logs:`
[2020-10-29 00:29:20,852] INFO Refusing session request for client /10.244.4.86:47054 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
`FULL LOGS:` [node-0-log.log](https://github.com/confluentinc/common/files/5465751/node-0-log.log)
`zookeeper-node-1 logs:`
244.3.185:36040 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer).
`FULL LOGS:` [node-1-log.log](https://github.com/confluentinc/common/files/5465752/node-1-log.log)
`zookeeper-node-2 logs:`
[2020-10-29 00:28:59,554] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
FULL LOGS: [node-2-log.log](https://github.com/confluentinc/common/files/5465754/node-2-log.log)
although the zookeeper cluster seems to be up, I'm not able to get anything from it. If I try to use cli to list a path I'm getting a timeout.
Any Ideas?
If this is not the place for this issue please guide me.
thanks
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked node-0, node-1, and node-2 logs, focusing on the zxid mismatch and timeout messages after repeated restarts. Review the described Kubernetes deployment and Confluent Docker image configuration to determine why a rejoined node cannot serve sessions. Done means identifying a reproducible cause and documenting a verified recovery or configuration change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, kubernetes
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100