Controller status is either running or CrashLoopBackOff and invoker waiting to initialize

Im unable to get the Controller and Invoker to running 1/1 state.

```
$kubectl get pods -n openwhisk
NAME READY STATUS RESTARTS AGE
owdev-alarmprovider-6c89d8846b-rjpqf 1/1 Running 0 95d
owdev-apigateway-d6d756db8-x7kb4 1/1 Running 0 95d
owdev-controller-0 0/1 Running 15 3h29m
owdev-couchdb-5666bfc5f6-gnktm 1/1 Running 0 95d
owdev-gen-certs-649zd 0/1 Completed 0 95d
owdev-init-couchdb-f5g6k 0/1 Completed 0 95d
owdev-install-packages-7psb4 0/1 Completed 0 95d
owdev-invoker-0 0/1 Init:0/1 0 4h1m
owdev-kafka-0 1/1 Running 0 4h10m
owdev-kafkaprovider-7867778f74-rhhzt 1/1 Running 0 95d
owdev-nginx-67b8974c77-dcwt4 1/1 Running 0 95d
owdev-redis-cdb68fc8f-tdczh 1/1 Running 0 95d
owdev-wskadmin 1/1 Running 0 95d
owdev-zookeeper-0 1/1 Running 0 4h10m

$ kubectl logs owdev-controller-0 -n openwhisk
[2022-02-14T06:53:43.612Z] [INFO] Slf4jLogger started
[2022-02-14T06:53:44.040Z] [WARN] Failed to attach the instrumentation because the Kamon Bundle is not present on the classpath
[2022-02-14T06:53:44.196Z] [INFO] Started the Kamon StatsD reporter
[2022-02-14T06:53:44.504Z] [INFO] [#tid_sid_unknown] [Config] environment set value for limits.triggers.fires.perMinute
[2022-02-14T06:53:44.505Z] [INFO] [#tid_sid_unknown] [Config] environment set value for limits.actions.sequence.maxLength
[2022-02-14T06:53:44.505Z] [INFO] [#tid_sid_unknown] [Config] environment set value for limits.actions.invokes.concurrent
[2022-02-14T06:53:44.505Z] [INFO] [#tid_sid_unknown] [Config] environment set value for limits.actions.invokes.perMinute
[2022-02-14T06:53:44.506Z] [INFO] [#tid_sid_unknown] [Config] environment set value for runtimes.manifest
[2022-02-14T06:53:44.506Z] [INFO] [#tid_sid_unknown] [Config] environment set value for kafka.hosts
[2022-02-14T06:53:44.506Z] [INFO] [#tid_sid_unknown] [Config] environment set value for port
[2022-02-14T06:53:45.186Z] [INFO] [#tid_sid_unknown] [KafkaMessagingProvider] completed0 already exists and the user can see it, skipping creation.
[2022-02-14T06:53:45.366Z] [INFO] [#tid_sid_unknown] [KafkaMessagingProvider] health already exists and the user can see it, skipping creation.
[2022-02-14T06:53:45.534Z] [INFO] [#tid_sid_unknown] [KafkaMessagingProvider] cacheInvalidation already exists and the user can see it, skipping creation.
[2022-02-14T06:53:45.678Z] [INFO] [#tid_sid_unknown] [KafkaMessagingProvider] events already exists and the user can see it, skipping creation.
[2022-02-14T06:53:45.871Z] [INFO] [#tid_sid_controller] [Controller] starting controller instance 0 [marker:controller_startup0_counter:1393]
[2022-02-14T06:53:46.527Z] [INFO] [#tid_sid_dispatcher] [MessageFeed] handler capacity = 128, pipeline fill at = 128, pipeline depth = 256
[2022-02-14T06:53:46.718Z] [INFO] [#tid_sid_loadbalancer] [ShardingContainerPoolBalancerState] managedFraction = 0.9, blackboxFraction = 0.1
[2022-02-14T06:53:46.782Z] [INFO] [#tid_sid_dispatcher] [MessageFeed] handler capacity = 128, pipeline fill at = 128, pipeline depth = 256
[2022-02-14T06:53:47.131Z] [INFO] [#tid_sid_loadbalancer] [WhiskAction] [GET] serving from datastore: CacheKey(whisk.system/invokerHealthTestAction0) [marker:database_cacheMiss_counter:2654]
[2022-02-14T06:53:47.183Z] [INFO] [#tid_sid_loadbalancer] [CouchDbRestStore] [GET] 'test_whisks' finding document: 'id: whisk.system/invokerHealthTestAction0' [marker:database_getDocument_start:2706]
[2022-02-14T06:53:47.376Z] [INFO] [#tid_sid_loadbalancer] [CouchDbRestStore] [marker:database_getDocument_finish:2899:192]
[2022-02-14T06:53:47.427Z] [INFO] [#tid_sid_loadbalancer] [WhiskAction] invalidating CacheKey(whisk.system/invokerHealthTestAction0)
Exception in thread "main" org.apache.openwhisk.core.database.DocumentTypeMismatchException: document type class org.apache.openwhisk.core.entity.WhiskTrigger did not match expected type class org.apache.openwhisk.core.entity.WhiskAction.
[2022-02-14T06:53:47.432Z] [ERROR] [#tid_sid_loadbalancer] [InvokerPool] error creating test action for invoker health: org.apache.openwhisk.core.database.DocumentTypeMismatchException: document type class org.apache.openwhisk.core.entity.WhiskTrigger did not match expected type class org.apache.openwhisk.core.entity.WhiskAction.
at org.apache.openwhisk.core.database.StoreUtils$.deserialize(StoreUtils.scala:71)
at org.apache.openwhisk.core.database.CouchDbRestStore.$anonfun$get$5(CouchDbRestStore.scala:233)
at org.apache.openwhisk.core.database.CouchDbRestStore$$Lambda$1297/0000000058052500.apply(Unknown Source)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.Future$$Lambda$658/00000000A0007890.apply(Unknown Source)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.Promise$$Lambda$164/00000000091939C0.apply(Unknown Source)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
at akka.dispatch.BatchingExecutor$BlockableBatch$$Lambda$1131/000000007C0031D0.apply$mcV$sp(Unknown Source)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:836)

$ kubectl describe pod owdev-controller-0 -n openwhisk
Name: owdev-controller-0
Namespace: openwhisk
Priority: 0
Node: dvorak-2-3/10.1.0.23
Start Time: Sun, 13 Feb 2022 21:36:04 -0600
Labels: app=owdev-openwhisk
chart=openwhisk-1.0.1
controller-revision-hash=owdev-controller-c6c445889
heritage=Helm
name=owdev-controller
release=owdev
statefulset.kubernetes.io/pod-name=owdev-controller-0
Annotations:
Status: Running
IP: 10.244.1.227
IPs:
IP: 10.244.1.227
Controlled By: StatefulSet/owdev-controller
Init Containers:
wait-for-kafka:
Container ID: docker://6227eb6e184643a13a14256caf5f9a3c1c05f11c5300a4cc329013cc86772326
Image: openwhisk/ow-utils:3e6138d
Image ID: docker-pullable://openwhisk/ow-utils@sha256:0b1a3531ade6774069f3f5777784f1d2446d2165537dd617bbc7302bcfb690b1
Port:
Host Port:
Command:
sh
-c
cacert="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"; token="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"; while true; do rc=$(curl -sS --cacert $cacert --header "Authorization: Bearer $token" https://kubernetes.default.svc/api/v1/namespaces/openwhisk/endpoints/owdev-kafka | jq -r ".subsets[].addresses | length"); echo "num ready kafka endpoints is $rc"; if [ $rc -gt 0 ]; then echo "Success: ready kafka endpoint!"; break; fi; echo "kafka not ready yet; sleeping for 3 seconds"; sleep 3; done;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 13 Feb 2022 21:36:06 -0600
Finished: Sun, 13 Feb 2022 21:36:06 -0600
Ready: True
Restart Count: 0
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from owdev-core-token-4ptj9 (ro)
wait-for-couchdb:
Container ID: docker://eb91335d1e63d2eeb74bb37d3509f026fb2157d76e896a4c1bfb0e46ba1ec0d3
Image: busybox:latest
Image ID: docker-pullable://busybox@sha256:139abcf41943b8bcd4bc5c42ee71ddc9402c7ad69ad9e177b0a9bc4541f14924
Port:
Host Port:
Command:
sh
-c
while true; do echo 'checking CouchDB readiness'; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; if [ $result -eq 0 ]; then echo 'Success: CouchDB is ready!'; break; fi; echo '...not ready yet; sleeping 3 seconds before retry'; sleep 3; done;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 13 Feb 2022 21:36:07 -0600
Finished: Sun, 13 Feb 2022 21:36:07 -0600
Ready: True
Restart Count: 0
Environment:
READINESS_URL: http://owdev-couchdb.openwhisk.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from owdev-core-token-4ptj9 (ro)
Containers:
controller:
Container ID: docker://50bf9291af60ceaaa609157a376819b13469cc17ed3b71681033f2651ef8c51d
Image: openwhisk/controller:3e6138d
Image ID: docker-pullable://openwhisk/controller@sha256:51e0f338a942fe395358263cf5f76f673bb949b3d36494556fac760a22df899a
Ports: 8080/TCP, 2552/TCP, 19999/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/bash
-c
/init.sh `hostname | awk -F '-' '{print $NF}'`
State: Running
Started: Mon, 14 Feb 2022 00:53:41 -0600
Last State: Terminated
Reason: Error
Exit Code: 143
Started: Mon, 14 Feb 2022 00:40:21 -0600
Finished: Mon, 14 Feb 2022 00:53:40 -0600
Ready: False
Restart Count: 15
Liveness: http-get http://:8080/ping delay=200s timeout=600s period=200s #success=1 #failure=3
Readiness: http-get http://:8080/ping delay=200s timeout=600s period=200s #success=1 #failure=3
Environment:
PORT: 8080
TZ: UTC
CONFIG_whisk_info_date: Optional: false
CONFIG_whisk_info_buildNo: Optional: false
JAVA_OPTS: -Xmx1024M
CONTROLLER_OPTS:
RUNTIMES_MANIFEST: {
"runtimes": {
"nodejs": [
{
"kind": "nodejs:12",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v12",
"tag": "1.19.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "nodejs:14",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v14",
"tag": "1.19.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
},
"stemCells": [
{
"initialCount": 2,
"memory": "256 MB",
"reactive": {
"minCount": 1,
"maxCount": 4,
"ttl": "2 minutes",
"threshold": 1,
"increment": 1
}
}
]
}
],
"python": [
{
"kind": "python:3",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-python-v3.7",
"tag": "1.17.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
}
],
"swift": [
{
"kind": "swift:4.2",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v4.2",
"tag": "1.17.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "swift:5.1",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v5.1",
"tag": "1.17.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "swift:5.3",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v5.3",
"tag": "1.17.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "swift:5.4",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v5.4",
"tag": "1.17.0"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
}
],
"java": [
{
"kind": "java:8",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "java8action",
"tag": "1.17.0"
},
"deprecated": false,
"attached": {
"attachmentName": "jarfile",
"attachmentType": "application/java-archive"
},
"requireMain": true
}
],
"php": [
{
"kind": "php:7.3",
"default": false,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.3",
"tag": "1.17.0"
},
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "php:7.4",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.4",
"tag": "1.17.0"
},
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "php:8.0",
"default": false,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.4",
"tag": "1.17.0"
},
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
}
],
"ruby": [
{
"kind": "ruby:2.5",
"default": true,
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
},
"image": {
"prefix": "openwhisk",
"name": "action-ruby-v2.5",
"tag": "1.17.0"
}
}
],
"go": [
{
"kind": "go:1.15",
"default": true,
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
},
"image": {
"prefix": "openwhisk",
"name": "action-golang-v1.15",
"tag": "1.18.0"
}
}
],
"rust": [
{
"kind": "rust:1.34",
"default": true,
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
},
"image": {
"prefix": "openwhisk",
"name": "action-rust-v1.34",
"tag": "1.3.0"
}
}
],
"dotnet": [
{
"kind": "dotnet:2.2",
"default": true,
"deprecated": false,
"requireMain": true,
"image": {
"prefix": "openwhisk",
"name": "action-dotnet-v2.2",
"tag": "1.16.0"
},
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
},
{
"kind": "dotnet:3.1",
"default": false,
"deprecated": false,
"requireMain": true,
"image": {
"prefix": "openwhisk",
"name": "action-dotnet-v3.1",
"tag": "1.16.0"
},
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
}
],
"ballerina": [
{
"kind": "ballerina:0.990",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-ballerina-v0.990.2",
"tag": "nightly"
},
"deprecated": false,
"attached": {
"attachmentName": "codefile",
"attachmentType": "text/plain"
}
}
]
},
"blackboxes": [
{
"prefix": "openwhisk",
"name": "dockerskeleton",
"tag": "1.14.0"
}
]
}

LIMITS_ACTIONS_INVOKES_PERMINUTE: 60
LIMITS_ACTIONS_INVOKES_CONCURRENT: 30
LIMITS_TRIGGERS_FIRES_PERMINUTE: 60
LIMITS_ACTIONS_SEQUENCE_MAXLENGTH: 50
CONFIG_whisk_timeLimit_min: 100ms
CONFIG_whisk_timeLimit_max: 5m
CONFIG_whisk_timeLimit_std: 1m
CONFIG_whisk_memory_min: 128m
CONFIG_whisk_memory_max: 512m
CONFIG_whisk_memory_std: 256m
CONFIG_whisk_concurrencyLimit_min: 1
CONFIG_whisk_concurrencyLimit_max: 1
CONFIG_whisk_concurrencyLimit_std: 1
CONFIG_whisk_logLimit_min: 0m
CONFIG_whisk_logLimit_max: 10m
CONFIG_whisk_logLimit_std: 10m
CONFIG_whisk_activation_payload_max: 1048576
CONFIG_whisk_loadbalancer_blackboxFraction: 10%
CONFIG_whisk_loadbalancer_timeoutFactor: 2
KAFKA_HOSTS: owdev-kafka-0.owdev-kafka.openwhisk.svc.cluster.local:9092
CONFIG_whisk_kafka_replicationFactor:
CONFIG_whisk_kafka_topics_cacheInvalidation_retentionBytes:
CONFIG_whisk_kafka_topics_cacheInvalidation_retentionMs:
CONFIG_whisk_kafka_topics_cacheInvalidation_segmentBytes:
CONFIG_whisk_kafka_topics_completed_retentionBytes:
CONFIG_whisk_kafka_topics_completed_retentionMs:
CONFIG_whisk_kafka_topics_completed_segmentBytes:
CONFIG_whisk_kafka_topics_events_retentionBytes:
CONFIG_whisk_kafka_topics_events_retentionMs:
CONFIG_whisk_kafka_topics_events_segmentBytes:
CONFIG_whisk_kafka_topics_health_retentionBytes:
CONFIG_whisk_kafka_topics_health_retentionMs:
CONFIG_whisk_kafka_topics_health_segmentBytes:
CONFIG_whisk_kafka_topics_invoker_retentionBytes:
CONFIG_whisk_kafka_topics_invoker_retentionMs:
CONFIG_whisk_kafka_topics_invoker_segmentBytes:
CONFIG_whisk_couchdb_username: Optional: false
CONFIG_whisk_couchdb_password: Optional: false
CONFIG_whisk_couchdb_port: Optional: false
CONFIG_whisk_couchdb_protocol: Optional: false
CONFIG_whisk_couchdb_host: owdev-couchdb.openwhisk.svc.cluster.local
CONFIG_whisk_couchdb_provider: Optional: false
CONFIG_whisk_couchdb_databases_WhiskActivation: Optional: false
CONFIG_whisk_couchdb_databases_WhiskEntity: Optional: false
CONFIG_whisk_couchdb_databases_WhiskAuth: Optional: false
CONTROLLER_INSTANCES: 1
CONFIG_logback_log_level: INFO
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from owdev-core-token-4ptj9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
owdev-core-token-4ptj9:
Type: Secret (a volume populated by a Secret)
SecretName: owdev-core-token-4ptj9
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 8m15s (x45 over 3h24m) kubelet Readiness probe failed: Get "http://10.244.1.227:8080/ping": dial tcp 10.244.1.227:8080: connect: connection refused
Warning Unhealthy 2m10s (x47 over 3h25m) kubelet Liveness probe failed: Get "http://10.244.1.227:8080/ping": dial tcp 10.244.1.227:8080: connect: connection refused
```

I increased the initialDelaySeconds for the liveness and readiness probe for the controller, but it doesn't work.
Can you please tell me how to resolve this issue?

Start with the reported `kubectl get pods`, controller logs, and `kubectl describe pod` output for `owdev-controller-0` and `owdev-invoker-0`. Trace the `DocumentTypeMismatchException` involving `invokerHealthTestAction0` and the invoker init wait, then verify that both pods reach Ready 1/1 without restarts.

Written by the indexing model from the issue text.