confluentinc / confluentinc/confluent-kafka-python

AWS MSK broker certificate validation fails with custom CA

Open
#1,962 1 comment 4 reactions 0 assignees View on GitHub
component:librdkafka investigate further
Dominant language
Python
Stars
509
Forks
964
Avg merge
2d 2h
Merged PRs (30d)
14

Description

Steps to reproduce:
1. Deploy Amazon MSK instance 3.8.1 version.
2. Configure SASL_SSL security protocol.
3. Check the broker's certificate.
4. Download CA certificate that issued the broker's certificate.
5. Use this CA certificate in the consumer's SASL session configuration as `ssl.ca.location`.

Actual result:
SSL handshake failed: ssl/statem/statem_clnt.c:2103:tls_post_process_server_certificate error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 297ms in state SSL_HANDSHAKE) (_SSL)

Expected result:
SSL handshake successful, connection established

```
$ openssl verify -CAfile ~/Downloads/Amazon-RSA-2048-M03.pem /tmp/server.crt
/tmp/server.crt: OK
```

P.S. It might be a problem with `librdkafka`, but I have no idea how to reproduce it.

[full_debug.log](https://github.com/user-attachments/files/19607798/full_debug.log)
Log
```
2025-04-04T16:41:10.877332236Z %3|1743784870.877|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-2.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.co]: sasl_ssl://b-2.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 232ms in state SSL_HANDSHAKE)
2025-04-04T16:41:11.710793051Z %3|1743784871.710|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-1.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.co]: sasl_ssl://b-1.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 366ms in state SSL_HANDSHAKE)
2025-04-04T16:41:12.533172399Z %3|1743784872.532|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.co]: sasl_ssl://b-3.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 184ms in state SSL_HANDSHAKE)
2025-04-04T16:41:13.501485909Z %3|1743784873.500|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.co]: sasl_ssl://b-3.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 273ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)
2025-04-04T16:41:14.664694469Z %3|1743784874.664|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-1.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.co]: sasl_ssl://b-1.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 323ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)
2025-04-04T16:41:17.632499779Z %3|1743784877.632|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-2.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.co]: sasl_ssl://b-2.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 294ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)
```

Dockerfile to reproduce
```
FROM python:3.13-alpine

# Install wget (to download the certificate) and pip dependencies
RUN apk add --no-cache --repository=https://dl-cdn.alpinelinux.org/alpine/edge/community \
wget gcc libc-dev librdkafka-dev=2.8.0-r0 \
&& pip install confluent-kafka==2.8.2

# Download the Amazon RSA 2048 M03 certificate
RUN mkdir /certs && wget -O /certs/amazon-rsa2048-m03.pem \
"https://www.amazontrust.com/repository/Amazon-RSA-2048-M03.pem"

# Copy the consumer script into the container
COPY consumer.py /app/consumer.py
WORKDIR /app

CMD ["python", "consumer.py"]
```

consumer.py
```
from confluent_kafka import Consumer

config = {
'bootstrap.servers': "b-2.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096,"
"b-3.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096,"
"b-1.mycluster.h6a8um.c2.kafka.eu-west-1.amazonaws.com:9096",
'security.protocol': "SASL_SSL",
'sasl.mechanism': "SCRAM-SHA-512",
'sasl.username': "user",
'sasl.password': "REDACTED",
'ssl.ca.location': "/certs/amazon-rsa2048-m03.pem",
'group.id': "my-consumer-group",
'auto.offset.reset': 'earliest',
'debug': 'all',
}

consumer = Consumer(config)
consumer.subscribe(["my_topic"]) # Replace with your topic name

try:
while True:
msg = consumer.poll(1.0)
if msg is None:
continue
if msg.error():
print("Consumer error: {}".format(msg.error()))
continue
print('Received message: {}'.format(msg.value().decode('utf-8')))
except KeyboardInterrupt:
pass
finally:
consumer.close()
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.