aws / aws/aws-lambda-base-images
Performance degradation in Amazon Linux 2023 runtimes when loading SSL certificates
- Dominant language
- No language data
- Stars
- 777
- Forks
- 118
- PR merge metrics
- No merged PRs in 30d
Description
There is a noticeable degradation in performance regarding operations that load SLL certificates (like creating an https connection) in Lambda runtimes running on Amazon Linux 2023 in comparison to runtimes that run on Amazon Linux 2.
The issue can easily reproduced with the following python snippet:
```python
import json
import http.client
def lambda_handler(event, context):
http.client.HTTPSConnection("")
return { 'statusCode': 200, 'body': json.dumps('Hello from Lambda!') }
```
Executing the function with python 3.12 runtime (128M of allocated memory) results in an execution time similar to
```
REPORT RequestId: 990ddd6f-406c-43cb-a4e4-78382f102b07 Duration: 917.49 ms Billed Duration: 918 ms Memory Size: 128 MB Max Memory Used: 47 MB Init Duration: 127.28 ms
```
and in comparison the same but running with the python 3.11 runtime
```
REPORT RequestId: 9992728e-fa11-422c-876c-c4319883b55b Duration: 298.25 ms Billed Duration: 299 ms Memory Size: 128 MB Max Memory Used: 43 MB Init Duration: 142.99 ms
```
So creating an HTTPSConnection in Amazon Linux 2023 is ~2-3 times slower than on Amazon Linux 2.
---
To narrow it down the function calls made when creating a new HTTPSConnection are:
* create new default `ssl.SSLContext` with `ssl._create_default_https_context` (which is an alias to [ssl.create_default_context](https://docs.python.org/3/library/ssl.html#ssl.create_default_context))
* `ssl._create_default_https_context` then calls [ssl.SSLContext.set_default_verify_paths](https://docs.python.org/3/library/ssl.html#ssl.SSLContext.set_default_verify_paths)
* `ssl.SSLContext.set_default_verify_paths` is a wrapper around OpenSSL [SSL_CTX_set_default_verify_paths](https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_default_verify_paths.html) where all the certificate loading happens
If python's `ssl.get_default_verify_paths()` is to be trusted then the certificates should be loaded from
```
DefaultVerifyPaths(cafile='/etc/pki/tls/cert.pem', capath='/etc/pki/tls/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/etc/pki/tls/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/etc/pki/tls/certs')
```
both, `SSL_CERT_FILE`, and `SSL_CERT_DIR` are unset and `/etc/pki/tls/certs` only contains
```
lrwxrwxrwx 1 root root 49 Feb 16 16:19 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx 1 root root 55 Feb 16 16:19 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
```
this seems to be pretty much identical in Amazon Linux 2023 and Amazon Linux 2.
the number of certificates in `/etc/ssl/crets` however is different with Amazon Linux 2 only containing only the two above certificates but Amazon Linux 2023 containing ~400. not sure though if (or how) these certificates are considered.
Note, this issue doesn't only happen in the python runtime but the degradation in performance can be observed in other runtimes (at least for Node 20) and for external Lambda extensions (e.g. some go extension that uses/creates an https connection) as well.
Contributor guide
Assessment
This issue has not been assessed yet.