PEM data was not found in buffer with valid RSA key in v1.31.0

Description

The OCI Cloud Controller Manager (CCM) v1.31.0 fails to initialize with "PEM data was not found in buffer" despite a valid RSA private key being provided at the specified path. The key is readable by the CCM process (confirmed via pre-start debug), works with the OCI CLI, and matches the fingerprint in the config. Tried with standard /etc/oci in-container mount and custom /oci-config mount. Tried regular PEM key and converted RSA key, always the same error from the cloud controller container. Tried using multiple different versions of CCM.

Environment

• CCM Version: ghcr.io/oracle/cloud-provider-oci:v1.31.0
• Kubernetes: v1.31.0
• OCI Region: us-ashburn-1
• Date: March 21, 2025

Steps to Reproduce

1. Generate an RSA key:
   bash
   openssl rsa -in /root/.oci/richard.tylka.clean.pem -out /root/.oci/richard.tylka.rsa.pem -traditional
   kubectl create secret generic oci-credential
   --from-file=cloud-provider.yaml
   --from-file=key.pem=/root/.oci/richard.tylka.rsa.pem
   -n kube-system

cloud-provider.yaml:

auth:
region: us-ashburn-1
tenancy: ocid1.tenancy.oc1..aaaaaaaa6bpzdsduwbti65kqm7gvubwvrspi7626cxona2t6fgguxn3ftvxa
user: ocid1.user.oc1..aaaaaaaaahcwxw3yxfayfxm666prjwi4667zhvnh4ddz5sdzclar3tjpww5q
key: /oci-config/key.pem
fingerprint: 76:75:60:81:15:20:95:b6:3a:cb:29:c6:38:ad:83:76
compartment: ocid1.compartment.oc1..aaaaaaaameevqllpkruwsepv2ps3otacm7vgx257uoi2o7sgeischov64hja
vcn: ocid1.vcn.oc1.iad.amaaaaaausti56qaepofn65jytrp37n5o4jopfbvxnquomr2dt46khs24s4a

oci-ccm.yaml:

oci-ccm.txt
kubectl logs -n kube-system -l k8s-app=oci-cloud-controller-manager -c oci-cloud-controller-manager --tail=-1
total 4
-rw-r--r-- 1 root root 1675 Mar 21 18:45 key.pem
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAtu3ghDp8/LUR6Jc2GNHsDvcFRUgjB72g0yNG9HmPxas785Bu
F6DD5p9WjtpyLuwh2/MH87eDPtJzdmVGX2huI9BS130G4f4e0sGqBGIT0XMhlL+a
9zUAANhZavRWOY6+9qNKdB+MhPMsw5W3Pqc1g4MOjlRBlYVUa60lHZk6OEyi2SqI
lUziYbwBBSQUH7xWkk6ObbH+kdSEabaGhBUvmQ6X+2cYdJAZyrF23L1vAHjmS26H
...truncated...
JU6F8NeWiPiqUQ/kAO8FmG/3gYCIA6WDy0WmyOj7OvG2cs9yKw2hzu7lHw9sk+4D
6qvRAoGAG0TXp5Ub+gpzXx1wQj6pvb8IQKyazROtXst4M0bsqYmrJgw/fwUNoNh9
6Ks0N2W5gSLme5C8Irxi1mZPwntd5/jbwpYdbDCUA7svevM3CclHf5xLQwJtdg+i
bruvA3e2A1Gw8fp29l9OCrIwNsIUQPt4yfRryWpX/VjBYJlzZ40=
-----END RSA PRIVATE KEY-----
2025-03-21T18:56:23.336Z INFO oci-cloud-controller-manager/main.go:69 oci-cloud-controller-manager {"version": "411bfeb2", "build": "411bfeb2"}
I0321 18:56:24.870569 1 serving.go:386] Generated self-signed cert in-memory
I0321 18:56:25.538715 1 serving.go:386] Generated self-signed cert in-memory
W0321 18:56:25.538742 1 client_config.go:659] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0321 18:56:26.555771 1 requestheader_controller.go:247] Loaded a new request header values for RequestHeaderAuthRequestController
2025-03-21T18:56:26.560Z INFO metadata/instance_metadata.go:76 Metadata endpoint /opc/v2/instance/ returned response successfully
2025-03-21T18:56:26.571Z INFO metadata/instance_metadata.go:76 Metadata endpoint /opc/v2/instance/ returned response successfully
2025-03-21T18:56:26.571Z INFO client/utils.go:95 OCI using read rate limit configuration: QPS=20, bucket=5 {"component": "cloud-controller-manager"}
2025-03-21T18:56:26.571Z INFO client/utils.go:99 OCI using write rate limit configuration: QPS=20, bucket=5 {"component": "cloud-controller-manager"}
F0321 18:56:26.571981 1 main.go:82] Cloud provider could not be initialized: could not init cloud provider "oci": NewComputeClientWithConfigurationProvider: can not create client, bad configuration: PEM data was not found in buffer

Additional Info
Key works with OCI CLI using the same fingerprint.

Tested with PKCS#8 and RSA formats, symlinks and real files, various paths (/etc/oci/key.pem, /oci-config/key.pem).

Debug container confirms key presence.

Full logs available upon request.

Suspected Cause
Bug in NewComputeClientWithConfigurationProvider failing to parse a valid RSA key.

Start at the cloud provider initialization reported at main.go:82 and trace NewComputeClientWithConfigurationProvider, using the supplied Kubernetes configuration and key-format reproduction steps. Done means the CCM initializes successfully with the valid RSA or PKCS#8 key instead of reporting “PEM data was not found in buffer.”