AWS CLI v2 embedded CA bundle does not validate a TLS chain validated by certifi and OpenSSL
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
### Describe the issue
## Summary
We are observing a TLS validation issue when using AWS CLI v2 against our S3-compatible HTTPS endpoint.
With the default AWS CLI v2 configuration, the command fails with a certificate validation error.
However, the same endpoint certificate chain is successfully validated by:
- the current `certifi` / Mozilla CA bundle;
- the default OpenSSL / operating system trust store.
Based on these observations, my current understanding is that the issue may be related to the CA bundle embedded in AWS CLI v2 / botocore, rather than to the endpoint certificate chain itself.
This does not appear to be a private or enterprise CA use case. The endpoint uses a public TLS certificate chain.
_Possible related issue :_ [#9017](https://github.com/aws/aws-cli/issues/9017)
## Environment
AWS CLI version: `aws-cli/2.34.58`
Operating system: `Ubuntu`
Endpoint: `https://custom.s3.server`
AWS CLI embedded CA bundle used by default:
```text
/usr/local/aws-cli/v2/current/dist/awscli/botocore/cacert.pem
```
## Current behavior
Running AWS CLI v2 against the endpoint fails with a TLS certificate validation error.
**command:**
```bash
aws s3 ls \
--endpoint-url "https://custom.s3.server" \
--debug
```
Error:
```text
botocore.exceptions.SSLError: SSL validation failed for https://custom.s3.server/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1081)
```
The debug output confirms that AWS CLI uses its embedded botocore CA bundle:
```text
/usr/local/aws-cli/v2/current/dist/awscli/botocore/cacert.pem
```
## Validation checks
### 1. AWS CLI embedded CA bundle
```bash
openssl s_client \
-connect "custom.s3.server:443" \
-servername "custom.s3.server" \
-CAfile "/usr/local/aws-cli/v2/current/dist/awscli/botocore/cacert.pem" \
-verify_return_error \
OK
OpenSSL / OS trust store -> OK
AWS CLI embedded botocore bundle -> FAILED
```
## Expected behavior
My expectation is that AWS CLI v2 should be able to validate a TLS certificate chain that is already trusted by current public CA trust stores, such as `certifi` / Mozilla and the default OpenSSL / OS trust store.
If this expectation is incorrect, I would appreciate clarification on the intended trust model for AWS CLI v2.
## Current workaround
The issue can be worked around by explicitly overriding the AWS CLI CA bundle.
For example:
```bash
export AWS_CA_BUNDLE=/path/to/valid/ca-bundle.pem
```
or:
```bash
aws ... --ca-bundle /path/to/valid/ca-bundle.pem
```
or:
```ini
[default]
ca_bundle = /path/to/valid/ca-bundle.pem
```
Using the OS CA bundle or the `certifi` CA bundle resolves the issue in our tests.
However, for a customer-facing usage, requiring users to manually configure a CA bundle is not ideal when the certificate chain is already trusted by standard public CA bundles.
## Request
If the embedded AWS CLI v2 / botocore CA bundle is expected to track public CA trust sources, could it be refreshed or reviewed so that it validates the same public TLS chains as current `certifi` / Mozilla and OpenSSL trust stores?
I may be missing some details about the AWS CLI v2 trust model, so any clarification would be very helpful.
Thank you for your help and for maintaining AWS CLI.
### Additional Information/Context
_No response_
### CLI version used
aws-cli/2.34.58
### Environment details (OS name and version, etc.)
Windows and Ubuntu 24.04.4
Contributor guide
Assessment
This issue has not been assessed yet.