Pulsar won't starting on kubernetes, some pods in init state
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
I am running on qemu (libvirt) vagrant k8s cluster with 1 master and 2 nodes with following config (just for imagination that it has enough resources):
```
DISK_COUNT ?= 1
DISK_SIZE_GB ?= 150
# VM Resources
MASTER_CPUS ?= 2
MASTER_MEMORY_SIZE_GB ?= 12
NODE_CPUS ?= 6
NODE_MEMORY_SIZE_GB ?= 32
NODE_COUNT ?= 2
```
I use following values file (customized from examples): https://gist.github.com/archenroot/c6c15b957758226473530825deae7649
I use following sequence to install pulsar on k8s (tls is disabled as there was some additional issue with webhook):
```
#!/usr/bin/env bash
helm repo add apache https://pulsar.apache.org/charts
helm repo update
#kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.4.0/cert-manager.yaml
#git clone https://github.com/apache/pulsar-helm-chart
cd pulsar-helm-chart
#sh scripts/cert-manager/install-cert-manager.sh
sh scripts/pulsar/prepare_helm_release.sh -n pulsar -k pulsar-mini -c \
--pulsar-superusers superadmin,proxy-admin,broker-admin,client-admin,admin
cd ..
rm -rf pulsar-helm-chart
helm install \
--values values-andromeda-local-cluster.yaml \
--namespace pulsar \
pulsar-mini apache/pulsar
```
During my tests I experienced also following error with replica set to 1 for bookkeeper, zookepier and broker:
`apache pulsar statefulsets.apps does no t implement the scale subresource on `
But at moment I have 2 replicas config (as per value file) and played bit with disabling enabling components and after about 15 minutes pulsar namespace looks like this:
zangetsu@andromeda ~ $ kubectl get pods -n pulsar
```
NAME READY STATUS RESTARTS AGE
pulsar-mini-broker-0 0/1 Init:0/2 0 13m
pulsar-mini-proxy-0 0/1 Init:0/2 0 8m16s
pulsar-mini-pulsar-init-d6r46 0/1 Init:0/3 0 14m
pulsar-mini-pulsar-manager-6c6889dff-brggz 1/1 Running 0 8m16s
pulsar-mini-toolset-0 1/1 Running 0 8m16s
pulsar-mini-zookeeper-0 1/1 Running 0 30m
```
kubectl describe for all pods in Init state here:
```
zangetsu@andromeda ~ $ gh gist create pulsar-mini-broker-0.pod
- Creating gist pulsar-mini-broker-0.pod
✓ Created gist pulsar-mini-broker-0.pod
https://gist.github.com/9ca2ee545aaae46c3a689dcd7c70f53d
zangetsu@andromeda ~ $ gh gist create pulsar-mini-proxy-0.pod
- Creating gist pulsar-mini-proxy-0.pod
✓ Created gist pulsar-mini-proxy-0.pod
https://gist.github.com/fdcb20259cd58eb02b5541feace9694e
zangetsu@andromeda ~ $ gh gist create pulsar-mini-pulsar-init-d6r46.pod
- Creating gist pulsar-mini-pulsar-init-d6r46.pod
✓ Created gist pulsar-mini-pulsar-init-d6r46.pod
https://gist.github.com/c08a7dab67f601a7289bc7338d20d07a
```
**Expected behavior**
Pulsar is up and running...
**Screenshots**
Pods in octant

StatefulSets

**Desktop (please complete the following information):**
- OS: [e.g. iOS] Gentoo, but workers in qemu running Centos8
**Additional context**
Add any other context about the problem here.
```
zangetsu@andromeda ~ $ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:52:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
```
I am bit lost about where to look for possible issue
Contributor guide
Research direction
Start with values-andromeda-local-cluster.yaml and scripts/pulsar/prepare_helm_release.sh, then inspect the linked pod descriptions for the broker, proxy, and pulsar-init pods. Reproduce the Helm installation with the supplied Kubernetes 1.21 setup and identify why the init containers remain pending; done means the affected pods complete initialization and Pulsar is running.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes
- Domain
- devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100