aws / aws/aws-cli

AWS CLI v2 embedded CA bundle does not validate a TLS chain validated by certifi and OpenSSL

Open
#10,389 3 comments 1 reaction 1 assignee Claimed by @RyanFitzSimmonsAK View on GitHub
feature-request p3 source-distribution
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

### Describe the issue

## Summary

We are observing a TLS validation issue when using AWS CLI v2 against our S3-compatible HTTPS endpoint.

With the default AWS CLI v2 configuration, the command fails with a certificate validation error.

However, the same endpoint certificate chain is successfully validated by:

- the current `certifi` / Mozilla CA bundle;
- the default OpenSSL / operating system trust store.

Based on these observations, my current understanding is that the issue may be related to the CA bundle embedded in AWS CLI v2 / botocore, rather than to the endpoint certificate chain itself.

This does not appear to be a private or enterprise CA use case. The endpoint uses a public TLS certificate chain.

_Possible related issue :_ [#9017](https://github.com/aws/aws-cli/issues/9017)

## Environment

AWS CLI version: `aws-cli/2.34.58`

Operating system: `Ubuntu`

Endpoint: `https://custom.s3.server`

AWS CLI embedded CA bundle used by default:

```text
/usr/local/aws-cli/v2/current/dist/awscli/botocore/cacert.pem
```

## Current behavior

Running AWS CLI v2 against the endpoint fails with a TLS certificate validation error.

**command:**

```bash
aws s3 ls \
--endpoint-url "https://custom.s3.server" \
--debug
```

Error:

```text
botocore.exceptions.SSLError: SSL validation failed for https://custom.s3.server/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1081)
```

The debug output confirms that AWS CLI uses its embedded botocore CA bundle:

```text
/usr/local/aws-cli/v2/current/dist/awscli/botocore/cacert.pem
```

## Validation checks

### 1. AWS CLI embedded CA bundle

```bash
openssl s_client \
-connect "custom.s3.server:443" \
-servername "custom.s3.server" \
-CAfile "/usr/local/aws-cli/v2/current/dist/awscli/botocore/cacert.pem" \
-verify_return_error \
OK
OpenSSL / OS trust store -> OK
AWS CLI embedded botocore bundle -> FAILED
```

## Expected behavior

My expectation is that AWS CLI v2 should be able to validate a TLS certificate chain that is already trusted by current public CA trust stores, such as `certifi` / Mozilla and the default OpenSSL / OS trust store.

If this expectation is incorrect, I would appreciate clarification on the intended trust model for AWS CLI v2.

## Current workaround

The issue can be worked around by explicitly overriding the AWS CLI CA bundle.

For example:

```bash
export AWS_CA_BUNDLE=/path/to/valid/ca-bundle.pem
```

or:

```bash
aws ... --ca-bundle /path/to/valid/ca-bundle.pem
```

or:

```ini
[default]
ca_bundle = /path/to/valid/ca-bundle.pem
```

Using the OS CA bundle or the `certifi` CA bundle resolves the issue in our tests.

However, for a customer-facing usage, requiring users to manually configure a CA bundle is not ideal when the certificate chain is already trusted by standard public CA bundles.

## Request

If the embedded AWS CLI v2 / botocore CA bundle is expected to track public CA trust sources, could it be refreshed or reviewed so that it validates the same public TLS chains as current `certifi` / Mozilla and OpenSSL trust stores?

I may be missing some details about the AWS CLI v2 trust model, so any clarification would be very helpful.

Thank you for your help and for maintaining AWS CLI.

### Additional Information/Context

_No response_

### CLI version used

aws-cli/2.34.58

### Environment details (OS name and version, etc.)

Windows and Ubuntu 24.04.4

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.