kudobuilder / kudobuilder/operators
[ZK] Triggering validation plan returns an error for zookeeper operator
- Dominant language
- Shell
- Stars
- 237
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
### Description
The `validation` plan of zookeeper operator does not run properly and marked as `COMPLETED`. Please find the detailed logs below.
```
└── zookeeper-instance (Operator-Version: "zookeeper-3.4.14-0.3.1" Active-Plan: "validation")
├── Plan deploy (serial strategy) [NOT ACTIVE]
│ ├── Phase zookeeper (parallel strategy) [NOT ACTIVE]
│ │ └── Step deploy [NOT ACTIVE]
│ └── Phase validation (serial strategy) [NOT ACTIVE]
│ ├── Step validation [NOT ACTIVE]
│ └── Step cleanup [NOT ACTIVE]
├── Plan not-allowed (serial strategy) [NOT ACTIVE]
│ └── Phase not-allowed (serial strategy) [NOT ACTIVE]
│ └── Step not-allowed [NOT ACTIVE]
└── Plan validation (serial strategy) [COMPLETE], last updated 2021-01-04 20:10:40
└── Phase connection (serial strategy) [COMPLETE]
├── Step connection [COMPLETE]
└── Step cleanup [COMPLETE]
```
#### Command
```
kubectl kudo plan trigger --name=validation --instance=zookeeper-instance
```
The `kudo-controller` logs are flooded with
```
2021/01/04 14:20:10 HealthUtil: unknown type *v1beta1.PodDisruptionBudget is marked healthy by default
2021/01/04 14:20:10 HealthUtil: statefulset "zookeeper-instance-zookeeper" is not healthy: Waiting for 1 pods to be ready...
2021/01/04 14:20:10 TaskExecution: object default/zookeeper-instance-zookeeper is NOT healthy: statefulset "zookeeper-instance-zookeeper" is not healthy: Waiting for 1 pods to be ready...
2021/01/04 14:20:10 PlanExecution: 'deploy' step(s) (instance: default/zookeeper-instance) of the deploy.zookeeper are not ready
2021/01/04 14:20:10 InstanceController: Received Reconcile request for instance default/zookeeper-instance
```
The plan is supposed to trigger a job which in turn will print the zookeeper URI. But it is unable to create any job stating
```
HealthUtil: job "zookeeper-instance-validation" still running or failed
2021/01/04 14:20:28 TaskExecution: object default/zookeeper-instance-validation is NOT healthy: job "zookeeper-instance-validation" still running or failed
2021/01/04 14:20:28 PlanExecution: 'validation' task(s) (instance: default/zookeeper-instance) of the deploy.validation.validation are not ready
2021/01/04 14:20:28 PlanExecution: 'validation,cleanup' step(s) (instance: default/zookeeper-instance) of the deploy.validation are not ready
```
The `zookeeper-instance` StatefulSet looks to be okay.
```
""2021-01-04 14:24:16,272 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@222] - Accepted socket connection from /127.0.0.1:39720
""2021-01-04 14:24:16,272 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@908] - Processing ruok command from /127.0.0.1:39720
""2021-01-04 14:24:16,273 [myid:3] - INFO [Thread-290:NIOServerCnxn@1056] - Closed socket connection for client /127.0.0.1:39720 (no session established for client)
```
Lastly, I am getting a TLS handshake error as well
```
2021/01/04 14:20:31 InstanceController: Error when updating instance status. Operation cannot be fulfilled on instances.kudo.dev "zookeeper-instance": the object has been modified; please apply your changes to the latest version and try again
2021/01/04 14:20:32 InstanceController: Received Reconcile request for instance default/zookeeper-instance
2021/01/04 14:20:32 Computing health out of 0 Deployments, 0 ReplicaSets, 1 StatefulSets, 0 DaemonSets, 3 Pods
2021/01/04 14:20:32 Updating instance default/zookeeper-instance readiness to: true
2021/01/04 14:20:32 InstanceController: Readiness did not change for default/zookeeper-instance. Not updating.
2021/01/04 14:20:32 http: TLS handshake error from 10.0.130.81:56732: EOF
2021/01/04 14:20:42 http: TLS handshake error from 10.0.130.81:56844: EOF
...
```
#### KUDO Version
```
KUDO Version: version.Info{GitVersion:"0.17.2", GitCommit:"d902714c", BuildDate:"2020-11-16T20:34:11Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64", KubernetesClientVersion:"v0.19.2"}
```
I tried this with KUDO version `0.17.0` and was getting the same error.
Contributor guide
Research direction
Start by reproducing the failure with `kubectl kudo plan trigger --name=validation --instance=zookeeper-instance` and inspect the validation plan and kudo-controller logs. Trace why the validation job is reported as still running or failed despite the StatefulSet being ready; done means the validation plan creates and completes its job and prints the ZooKeeper URI without the reported errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100