OW 1.1.0 Unable to gathering actions/activation logs on OKD/OpenShift 4.5

**Environment:**
OKD 4.5

**Issue Description:**
I am trying to deploy OW on multi-node OKD 4.5 cluster 3 master node and 3 compute node with support to Dynamic Volume Provision (custom StorageClass)

**Other Installation Details:**
I followed the official guide [here ](https://github.com/apache/openwhisk-deploy-kube)

```
oc label node openwhisk-role=invoker

oc adm policy add-scc-to-user anyuid -z default
oc adm policy add-scc-to-user privileged -z default
oc adm policy add-scc-to-user anyuid -z openwhisk-core
oc adm policy add-scc-to-user privileged -z openwhisk-core
oc adm policy add-scc-to-user anyuid -z owdev-init-sa
oc adm policy add-scc-to-user privileged -z owdev-init-sa
```

Deploying to OKD/OpenShift uses the command sequence:

```
helm template owdev ./helm/openwhisk -n openwhisk -f _mycluster.yaml_ > owdev.yaml
oc apply -f owdev.yaml
```

_mycluster.yaml_

```
k8s:
persistence:
enabled: true
hasDefaultStorageClass: false
explicitStorageClass: ceph-fs

whisk:
ingress:
type: OpenShift
apiHostName: openwhisk.apps.lab.okd.
apiHostPort: 443
apiHostProto: https
domain: openwhisk.apps.lab.okd.
testing:
includeTests: false

invoker:
#Tried with and without this option, but no gathering actions/activation logs
options: "-Dwhisk.spi.LogStoreProvider=org.apache.openwhisk.core.containerpool.logging.LogDriverLogStoreProvider"
containerFactory:
impl: kubernetes
```

I have also tried with:

```
options: "-Dwhisk.spi.LogStoreProvider=org.apache.openwhisk.core.containerpool.logging.SplunkLogStoreProvider"

Invoker failed with this error:

[2021-01-09T15:42:18.166Z] [ERROR] [#tid_sid_invoker] [Invoker] Failed to initialize reactive invoker: Cannot convert configuration to a org.apache.openwhisk.core.containerpool.logging.SplunkLogStoreConfig. Failures are:
at 'whisk.logstore':
- (jar:file:/invoker/lib/openwhisk-common-1.0.0-SNAPSHOT.jar!/application.conf:505) Key not found: 'splunk'.
```

oc get pods

```
NAME READY STATUS RESTARTS AGE
owdev-alarmprovider-6f55fd9bd-s9mlz 1/1 Running 0 13m
owdev-apigateway-5d84c5969d-tg9qx 1/1 Running 0 13m
owdev-controller-0 1/1 Running 0 13m
owdev-couchdb-56885dd9d7-qhwnq 1/1 Running 0 13m
owdev-invoker-0 1/1 Running 0 13m
owdev-kafka-0 1/1 Running 0 13m
owdev-kafkaprovider-dfb77c74d-h8xtk 1/1 Running 0 13m
owdev-redis-7d46bc659-ktnz5 1/1 Running 0 13m
owdev-wskadmin 1/1 Running 0 13m
owdev-zookeeper-0 1/1 Running 0 13m
wskowdev-invoker-00-1-prewarm-nodejs10 1/1 Running 0 7m57s
wskowdev-invoker-00-2-prewarm-nodejs10 1/1 Running 0 7m57s
wskowdev-invoker-00-3-whisksystem-invokerhealthtestaction0 1/1 Running 0 7m56s
wskowdev-invoker-00-4-guest-hello-world 1/1 Running 0 3m32s
wskowdev-invoker-00-5-prewarm-nodejs10 1/1 Running 0 55s
```

**Issue Detail:**

fixed_params.yaml

```
packages:
hello_world_package:
version: 1.0
license: Apache-2.0
actions:
hello_world:
function: hello.js
runtime: nodejs:14
inputs:
name: Unreal
```

hello.js

```
function main(args) {
const name = args && args.name || "stranger";
const greeting = `Hello ${name}!`
console.log(greeting);
return { greeting: greeting };
}
```

Deploy

`wskdeploy -m fixed_params.yaml`

Run Action

```
wsk -i action invoke /guest/hello_world_package/hello_world

ok: invoked /guest/hello_world_package/hello_world with id f64a401d53d841f58a401d53d841f540
```

Verify Activation

```
wsk activation list

Datetime Activation ID Kind Start Duration Status Entity
2021-01-09 14:36:09 f64a401d53d841f58a401d53d841f540 nodejs:10 cold 90ms success guest/hello_world:0.0.2
```

Get Activation by ID

```
wsk activation get f64a401d53d841f58a401d53d841f540

ok: got activation f64a401d53d841f58a401d53d841f540
{
"namespace": "guest",
"name": "hello_world",
"version": "0.0.2",
"subject": "guest",
"activationId": "f64a401d53d841f58a401d53d841f540",
"start": 1610199369225,
"end": 1610199369315,
"duration": 90,
"statusCode": 0,
"response": {
"status": "success",
"statusCode": 0,
"success": true,
"result": {
"greeting": "Hello Unreal!"
}
},
**"logs": [],**
"annotations": [
{
"key": "path",
"value": "guest/hello_world_package/hello_world"
},
{
"key": "waitTime",
"value": 251
},
{
"key": "kind",
"value": "nodejs:10"
},
{
"key": "timeout",
"value": false
},
{
"key": "limits",
"value": {
"concurrency": 1,
"logs": 10,
"memory": 256,
"timeout": 60000
}
},
{
"key": "initTime",
"value": 81
}
],
"publish": false
}
```

Verify only logs

```
wsk activation logs f64a401d53d841f58a401d53d841f540

```

Verify Only result

```
wsk activation result f64a401d53d841f58a401d53d841f540

{
"greeting": "Hello Unreal!"
}
```

Verify invoker pod logs (before it is removed)

```
oc logs wskowdev-invoker-00-4-guest-hello-world

Hello Unreal!
XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX

```
_I have also read the entire database with wskadmin db get whisks with --docs to include the full documents.
No logs entry are present in DB._

**Conclusion**

Everything seems to work correctly but the logs are not collected with any language (tested also python and java).
The exact same test appears to work fine on ow released with docker compose on a single VM
It appears to be an ow bug on OKD / OpenShift.

**Questions:**

Which component of OW is responsible for collecting the invoker / action / activation logs?
Useful advice on how to try to solve?

Start with the Helm deployment values in mycluster.yaml and the invoker logging options, then compare the OpenShift deployment with the Docker Compose setup. Use oc logs on the invoker pod and wsk activation get/logs to trace where action output is lost, and inspect the stored activation document with wskadmin db get whisks. Done means logs are collected and returned for activations on OKD/OpenShift.

Written by the indexing model from the issue text.