confluentinc / confluentinc/common

Zookeeper throws timeout after several restarts of nodes

Open
#304 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17
Forks
243
Avg merge
1d 21h
Merged PRs (30d)
38

Description

I have a problem with zookeeper deployed on kubernetes using Confluentinc Docker Images.

### Cluster setup: ###
- 3 nodes in digital ocean ( 4 CPU, 8 GB or RAM) as kafka dedicated nodepool. ( Only zookeeper and kafka are running here, and some monitoring daemon set).

- If a zookeeper node/pod ( any node) restarts several times the zookeeper cluster is not accessible anymore, and to make it work I have to redeploy all the nodes.

- Once the node rejoined the cluster, all zookeeper nodes start to show this kind of error:
Each node has one kafka pod and one zookeeper pod.

### Errors one each node: ###
`zookeeper-node-0 logs:`
[2020-10-29 00:29:20,852] INFO Refusing session request for client /10.244.4.86:47054 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
`FULL LOGS:` [node-0-log.log](https://github.com/confluentinc/common/files/5465751/node-0-log.log)

`zookeeper-node-1 logs:`
244.3.185:36040 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer).
`FULL LOGS:` [node-1-log.log](https://github.com/confluentinc/common/files/5465752/node-1-log.log)

`zookeeper-node-2 logs:`
[2020-10-29 00:28:59,554] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
FULL LOGS: [node-2-log.log](https://github.com/confluentinc/common/files/5465754/node-2-log.log)

although the zookeeper cluster seems to be up, I'm not able to get anything from it. If I try to use cli to list a path I'm getting a timeout.
Any Ideas?

If this is not the place for this issue please guide me.
thanks

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked node-0, node-1, and node-2 logs, focusing on the zxid mismatch and timeout messages after repeated restarts. Review the described Kubernetes deployment and Confluent Docker image configuration to determine why a rejoined node cannot serve sessions. Done means identifying a reproducible cause and documenting a verified recovery or configuration change.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, kubernetes
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.