TektonConfig never gets a ready state preventing Openshift Pipelines installation (OpenShift 4.12)

Expected Behavior

Have Openshift Pipelines operator installed succesfully

Actual Behavior

Openshif Pipelines operator never gets a succesfull state

Steps to Reproduce the Problem

This issue happens intermitently, unfortunately. The steps to reproduce it is really just install latest Openshift Pipelines (channel is currently using openshift-pipelines-operator-rh.v1.11.0bundle).

Additional Info

• Openshift 4.12

  Output of oc version:

Client Version: 4.10.51
Server Version: 4.12.16
Kubernetes Version: v1.25.8+37a9a08

Tekton Pipeline version

Output of tkn version:

Client version: 0.30.1
Pipeline version: v0.47.2
Triggers version: v0.24.1
Operator version: v0.67.0

Output of kubectl get pods -n openshift-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

v0.47.2                            

Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

Initial analysis of the problem

Secret tekton-triggers-core-interceptors-certs, in openshift-pipelines namespace, is created empty:

kind: Secret
apiVersion: v1
metadata:
  annotations:
    operator.tekton.dev/last-applied-hash: 690e03e6f63f2ea2c6aef2cb04bf95873bf0885667c29e58090578550868c439
  resourceVersion: '25482'
  name: tekton-triggers-core-interceptors-certs
  uid: 2c2b6f05-1fcf-4ff0-ac8a-375084ef4056
  creationTimestamp: '2023-06-26T22:42:35Z'
  namespace: openshift-pipelines
  ownerReferences:
    - apiVersion: operator.tekton.dev/v1alpha1
      kind: TektonInstallerSet
      name: trigger-main-static-94w48
      uid: 953cb70b-1e8b-428a-8a53-c88ee1d86b5d
      controller: true
      blockOwnerDeletion: true
  labels:
    app.kubernetes.io/component: interceptors
    app.kubernetes.io/instance: default
    app.kubernetes.io/name: core-interceptors
    app.kubernetes.io/part-of: tekton-triggers
    operator.tekton.dev/operand-name: tektoncd-triggers
    triggers.tekton.dev/release: v0.24.1
type: Opaque

Compare this with secret created in an environment where pipeline operator was installed successfully:

kind: Secret
apiVersion: v1
metadata:
  annotations:
    operator.tekton.dev/last-applied-hash: 2816755d405fd00f1fc54e9fb9c00dc5cf5419b3148c051c154275831f6f607f
  resourceVersion: '104528'
  name: tekton-triggers-core-interceptors-certs
  uid: fd66ab4e-94a7-44d1-abe9-01699612c6d0
  creationTimestamp: '2023-06-26T21:56:54Z'
  namespace: openshift-pipelines
  ownerReferences:
    - apiVersion: operator.tekton.dev/v1alpha1
      kind: TektonInstallerSet
      name: trigger-main-static-dqnrd
      uid: 17fb2b7b-b739-4d6a-b789-4b3be7f9d07c
      controller: true
      blockOwnerDeletion: true
  labels:
    app.kubernetes.io/component: interceptors
    app.kubernetes.io/instance: default
    app.kubernetes.io/name: core-interceptors
    app.kubernetes.io/part-of: tekton-triggers
    operator.tekton.dev/operand-name: tektoncd-triggers
    triggers.tekton.dev/release: v0.24.1
data:
  ca-cert.pem: >-
    LS0tLS1…
  server-cert.pem: >-
    LS0tLS1CRUdJTiB...
  server-key.pem: >-
    LS0tLS1CR…
type: Opaque

Absence of the data in this secre prevents TektonTrigger reconciliation, which prevents TektonConfig to be installed successfully... then pipeline is not installed - issues causes some TLS handshake issues in tekton-triggers-core-interceptors Deployment's pod.

Important: we are installing Openshift PIpelines in IBM Cloud Red Hat Openshift Service cluster. We have been using this for a long time alreay, and this issue used to happen sometimes - once every two weeks perhaps - but recently these problems are getting more frequent.

Thoughts on why secret is creating without any data?

Start by comparing the empty and populated tekton-triggers-core-interceptors-certs Secrets and inspect TektonTrigger/TektonConfig reconciliation alongside the deployment's TLS handshake errors. Done means identifying why the Secret is created without data, preventing reconciliation, and verifying that TektonConfig reaches a ready state during installation.