EKSA vSphere package controller pod x509 certificate signed by unknown authority

**What happened**:

- Package controller pod on EKSA vSphere cluster failling with `X509 certificate signed by unknown authority` error.
- Tried re-installing the package controller [1] using below commands and no luck. we still notice same error from pod logs.

```
helm uninstall -n eksa-packages eks-anywhere-packages
eksctl anywhere install packagecontroller -f .yaml
```

package controller pod logs

```
pulling package bundle: fetch manifest: Get "https://public.ecr.aws/v2/eks-anywhere/eks-anywhere-packages-bundles/manifests/v1-29-latest": x509: certificate signed by unknown authority
```

Helm list command output

```
helm list --all-namespaces
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
eks-anywhere-packages eksa-packages 1 2024-03-26 18:25:30.769847675 +0000 UTC deployed eks-anywhere-packages-0.3.13-eks-a-60 v0.3.13-86cb2ba2e629eae21c79bca6bf78149e81f2527f
```

Checked cert-manager and validate no errors.

```
> k logs cert-manager-848f9994fc-txvt9 -n cert-manager
I0320 20:27:42.651183 1 controller.go:251] "cert-manager/controller/build-context: configured acme dns01 nameservers" nameservers=["10.96.0.10:53"]
W0320 20:27:42.651245 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0320 20:27:42.653564 1 controller.go:72] "cert-manager/controller: enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]"
I0320 20:27:42.654030 1 controller.go:145] "cert-manager/controller: starting leader election"
I0320 20:27:42.654844 1 leaderelection.go:250] attempting to acquire leader lease kube-system/cert-manager-controller...
I0320 20:27:42.655194 1 controller.go:93] "cert-manager/controller: starting metrics server" address="[::]:9402"
I0320 20:27:42.655258 1 controller.go:138] "cert-manager/controller: starting healthz server" address="[::]:9403"
I0320 20:27:42.667318 1 leaderelection.go:260] successfully acquired lease kube-system/cert-manager-controller
I0320 20:27:42.668453 1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-ca"
I0320 20:27:42.668906 1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-acme"
I0320 20:27:42.669119 1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="gateway-shim"
I0320 20:27:42.670341 1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-selfsigned"
I0320 20:27:42.670348 1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-venafi"
I0320 20:27:42.671524 1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-vault"
I0320 20:27:42.671756 1 controller.go:215] "cert-manager/controller: starting controller" controller="ingress-shim"
I0320 20:27:42.671774 1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-vault"
I0320 20:27:42.671787 1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-key-manager"
I0320 20:27:42.671809 1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-issuing"
I0320 20:27:42.671820 1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-request-manager"
I0320 20:27:42.671831 1 controller.go:215] "cert-manager/controller: starting controller" controller="orders"
I0320 20:27:42.671842 1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-acme"
I0320 20:27:42.671868 1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-approver"
I0320 20:27:42.679670 1 controller.go:215] "cert-manager/controller: starting controller" controller="clusterissuers"
I0320 20:27:42.679696 1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-venafi"
I0320 20:27:42.679724 1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-ca"
I0320 20:27:42.679754 1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-selfsigned"
I0320 20:27:42.679772 1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-metrics"
I0320 20:27:42.679796 1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-readiness"
I0320 20:27:42.679816 1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-trigger"
I0320 20:27:42.679844 1 controller.go:215] "cert-manager/controller: starting controller" controller="issuers"
I0320 20:27:42.679860 1 controller.go:215] "cert-manager/controller: starting controller" controller="challenges"
I0320 20:27:42.679877 1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-revision-manager"
E0320 21:15:15.230396 1 controller.go:134] "cert-manager/issuers: issuer in work queue no longer exists" err="[issuer.cert-manager.io](http://issuer.cert-manager.io/) \"eks-anywhere-packages-selfsigned-issuer\" not found"
I0320 21:15:51.505607 1 conditions.go:203] Setting lastTransitionTime for Certificate "eks-anywhere-packages-serving-cert" condition "Ready" to 2024-03-20 21:15:51.505579818 +0000 UTC m=+2888.887189117
I0320 21:15:51.515170 1 conditions.go:96] Setting lastTransitionTime for Issuer "eks-anywhere-packages-selfsigned-issuer" condition "Ready" to 2024-03-20 21:15:51.515164279 +0000 UTC m=+2888.896773587
E0326 18:25:12.590651 1 controller.go:134] "cert-manager/issuers: issuer in work queue no longer exists" err="[issuer.cert-manager.io](http://issuer.cert-manager.io/) \"eks-anywhere-packages-selfsigned-issuer\" not found"
I0326 18:25:31.462353 1 conditions.go:203] Setting lastTransitionTime for Certificate "eks-anywhere-packages-serving-cert" condition "Ready" to 2024-03-26 18:25:31.462345478 +0000 UTC m=+511068.843954778
I0326 18:25:31.482040 1 conditions.go:96] Setting lastTransitionTime for Issuer "eks-anywhere-packages-selfsigned-issuer" condition "Ready" to 2024-03-26 18:25:31.482033776 +0000 UTC m=+511068.863643081
E0326 18:33:23.986091 1 controller.go:134] "cert-manager/issuers: issuer in work queue no longer exists" err="[issuer.cert-manager.io](http://issuer.cert-manager.io/) \"eks-anywhere-packages-selfsigned-issuer\" not found"
I0326 18:33:41.750271 1 conditions.go:203] Setting lastTransitionTime for Certificate "eks-anywhere-packages-serving-cert" condition "Ready" to 2024-03-26 18:33:41.750264124 +0000 UTC m=+511559.131873435
I0326 18:33:41.767384 1 conditions.go:96] Setting lastTransitionTime for Issuer "eks-anywhere-packages-selfsigned-issuer" condition "Ready" to 2024-03-26 18:33:41.767376284 +0000 UTC m=+511559.148985588
```

There are four secrets as Volume mount to Package controller pod

```
kubectl get secret -n eksa-packages webhook-server-cert -o yaml
kubectl get secret -n eksa-packages registry-mirror-cred -o yaml
ekubectl get secret -n eksa-packages ecr-token -o yaml
kubectl get secret -n eksa-packages aws-secret -o yaml
```

- `webhook-server-cert` secret got ca.crt, tls.crt and tls.key.crt files data in it. `ca.crt` shows 3 months validity for the certificate. `tls.key.crt` fails with below error when tried to read it.
- `ecr-token` got decoded using base64 and able to see json data needed to authenticate with ECR.

```
openssl x509 -in webhook-server-cert.tls.key.crt --noout --text

Could not read certificate from webhook-server-cert.tls.key.crt
800B5C3D057F0000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:crypto/store/store_result.c:151:
```

---

Docker pull works fine from Admin machine. below is output for manual docker pull.

```
aws sts get-caller-identity
{
"UserId": "REDACTED",
"Account": "REDACTED",
"Arn": "arn:aws:iam::REDACTED:user/service/eksa-curated-package-user"
}
aws ecr get-login-password | docker login --username AWS --password-stdin [REDACTED.dkr.ecr.us-west-2.amazonaws.com](http://REDACTED.dkr.ecr.us-west-2.amazonaws.com/)

Error response from daemon: login attempt to https://REDACTED.dkr.ecr.us-west-2.amazonaws.com/v2/ failed with status: 400 Bad Request
> aws ecr get-login-password | docker login --username AWS --password-stdin [REDACTED.dkr.ecr.us-east-1.amazonaws.com](http://REDACTED.dkr.ecr.us-east-1.amazonaws.com/)

WARNING! Your password will be stored unencrypted in /home/REDACTED/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

> docker pull [REDACTED.dkr.ecr.us-east-1.amazonaws.com/emissary-ingress/emissary:v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074](http://REDACTED.dkr.ecr.us-east-1.amazonaws.com/emissary-ingress/emissary:v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074)

v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074: Pulling from emissary-ingress/emissary
Digest: sha256:0429a4b17ea8b2845ec66de412640f599665aad52093ea62d5d564e788c9b5cc
Status: Image is up to date for [REDACTED.dkr.ecr.us-east-1.amazonaws.com/emissary-ingress/emissary:v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074](http://REDACTED.dkr.ecr.us-east-1.amazonaws.com/emissary-ingress/emissary:v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074)
[REDACTED.dkr.ecr.us-east-1.amazonaws.com/emissary-ingress/emissary:v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074](http://REDACTED.dkr.ecr.us-east-1.amazonaws.com/emissary-ingress/emissary:v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074)
```

---

References:
[1] https://anywhere.eks.amazonaws.com/docs/packages/packagecontroller/

**What you expected to happen**:
- `eksctl anywhere install` command should completed package controller installation correctly.

**How to reproduce it (as minimally and precisely as possible)**:
- Create a EKSA vSphere cluster. package controller should be installed by default. uninstall the package controller and re-install using below command,

```
helm uninstall -n eksa-packages eks-anywhere-packages
eksctl anywhere install packagecontroller -f .yaml
```

- Validate the installation timeouts

**Anything else we need to know?**:

**Environment**:
- EKS Anywhere Release: v0.19.0
- EKS Distro Release: Kubernetes version 1.29

Start by reproducing the package controller installation on an EKS Anywhere vSphere cluster using the documented helm uninstall and eksctl anywhere install packagecontroller commands. Inspect the package controller pod logs and mounted webhook-server-cert, ecr-token, registry-mirror-cred, and aws-secret secrets; done means the controller installs and pulls the package bundle without the x509 unknown-authority error.

Written by the indexing model from the issue text.