Karapace Schema Registry on GKE - unable to access Strimzi Kafka deployed on GKE

I have Karapace deployed on GKE, and the karapace pods have come up .. however, the connection to the Strimzi Kafka cluster is failing.

Note : Strimzi Kafka cluster is deployed for last 3 years and other applications are able to connect to the cluster from inside the Kubernetes cluster as well as outside the cluster. Also, Strimzi has self-signed certs and Karapace is not recognizing the certs causing SSL connection issues

Here is the pod logs :
```
karapace.coordinator.master_coordinator Thread-1 (_start_loop) WARNING Kafka client bootstrap failed.
karapace.kafka.admin.KafkaAdminClient schema-reader INFO Could not establish connection due to errors: {KafkaError{code=_SSL,val=-181,str="ssl://versa-kafka-poc-tf-kafka-external-bootstrap.kafka.svc.cluster.local:9094/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 3ms in state SSL_HANDSHAKE)"}, KafkaError{code=_ALL_BROKERS_DOWN,val=-187,str="1/1 brokers are down"}}
aiokafka Thread-1 (_start_loop) DEBUG Attempting to bootstrap via node at versa-kafka-poc-tf-kafka-external-bootstrap.kafka.svc.cluster.local:9094
aiokafka Thread-1 (_start_loop) ERROR Unable connect to "versa-kafka-poc-tf-kafka-external-bootstrap.kafka.svc.cluster.local:9094": [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)
karapace.coordinator.master_coordinator Thread-1 (_start_loop) WARNING Kafka client bootstrap failed.
aiokafka Thread-1 (_start_loop) DEBUG Attempting to bootstrap via node at versa-kafka-poc-tf-kafka-external-bootstrap.kafka.svc.cluster.local:9094
aiokafka Thread-1 (_start_loop) ERROR Unable connect to "versa-kafka-poc-tf-kafka-external-bootstrap.kafka.svc.cluster.local:9094": [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1007)
karapace.coordinator.master_coordinator Thread-1 (_start_loop) WARNING Kafka client bootstrap failed.
```

Here is the deployment yaml -
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: karapace-registry
namespace: ueba-karapace
spec:
replicas: 2
selector:
matchLabels:
app: karapace-registry
template:
metadata:
labels:
app: karapace-registry
spec:
initContainers:
- name: check-certs
image: busybox
command: ["/bin/sh", "-c"]
args:
- |
echo "Checking certificate files:"
ls -la /etc/karapace/cluster-certs/
ls -la /etc/karapace/certs/
echo "Certificate files exist? (1=yes, 0=no)"
echo "ca.p12: $([[ -f /etc/karapace/cluster-certs/ca.p12 ]] && echo 1 || echo 0)"
echo "ca.password: $([[ -f /etc/karapace/cluster-certs/ca.password ]] && echo 1 || echo 0)"
echo "user.p12: $([[ -f /etc/karapace/certs/user.p12 ]] && echo 1 || echo 0)"
echo "user.password: $([[ -f /etc/karapace/certs/user.password ]] && echo 1 || echo 0)"
echo "Testing DNS resolution:"
nslookup versa-kafka-poc-tf-kafka-external-bootstrap.kafka.svc.cluster.local
echo "Testing TCP connectivity to port 9094:"
nc -z -v versa-kafka-poc-tf-kafka-external-bootstrap.kafka.svc.cluster.local 9094 || echo "Failed with: $?"
echo "Init container checks completed."
volumeMounts:
- name: certificates
mountPath: /etc/karapace/certs/
readOnly: true
- name: cluster-certs
mountPath: /etc/karapace/cluster-certs/
readOnly: true
containers:
- name: karapace-registry
image: ghcr.io/aiven-open/karapace:latest
command: ["/bin/sh", "-c"]
args:
- |
# Create a custom config using p12 files
cat > /tmp/karapace.json << EOF
{
"bootstrap_uri": "versa-kafka-poc-tf-kafka-external-bootstrap.kafka.svc.cluster.local:9094",
"kafka_security_protocol": "SSL",
"security_protocol": "SSL",
"ssl_cafile": "/etc/karapace/cluster-certs/ca.crt",
"ssl_certfile": "/etc/karapace/certs/user.crt",
"ssl_keyfile": "/etc/karapace/certs/user.key",
"ssl_check_hostname": false,
"topic_name": "_schemas",
"host": "0.0.0.0",
"port": 8081,
"advertised_hostname": "karapace-registry-java",
"log_level": "DEBUG",
"master_eligibility": true,
"client_id": "karapace-registry-java",
"group_id": "karapace-registry",
"admin_metadata_max_age": 300,
"karapace_registry": true,
"karapace_rest": false
}
EOF

echo "Generated configuration:"
cat /tmp/karapace.json

# Start Karapace with the generated config
karapace /tmp/karapace.json
ports:
- containerPort: 8081
volumeMounts:
- name: certificates
mountPath: /etc/karapace/certs/
readOnly: true
- name: cluster-certs
mountPath: /etc/karapace/cluster-certs/
readOnly: true
resources:
requests:
memory: "256Mi"
cpu: "100m"
limits:
memory: "512Mi"
cpu: "500m"
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: KARAPACE_LOG_LEVEL
value: "DEBUG"
volumes:
- name: certificates
secret:
defaultMode: 420
secretName: karapace-sr-user
- name: cluster-certs
secret:
defaultMode: 420
secretName: versa-kafka-poc-tf-cluster-ca-cert
```

NOTE - the karapace.json config is updated based on the variables seen in the files ->

https://github.com/Aiven-Open/karapace/blob/8585479131e0b6c47f4e7c8282b7ff6a55266c08/src/karapace/core/kafka/common.py#L157

https://github.com/Aiven-Open/karapace/blob/8585479131e0b6c47f4e7c8282b7ff6a55266c08/src/karapace/core/config.py#L103

The karapace.json has the configurations I'm using. Also, the initContainer is added to check if the connectivity to the Kafka cluster is working, and that is ok.

Any ideas on how to fix this ?

tia!