Tests fail with openssl 3 due to test certificates not meeting current standards
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**Describe the issue**: When run against openssl 3, some of the tests that use the certificates in `distributed/tests` (`tls-cert.pem` et al) fail, with errors like "Client certificate validation failed: Basic Constraints of CA cert not marked critical". Per [this page](https://community.tenable.com/s/article/OpenSSL-change-in-Nessus-10-3-and-Nessus-Agent-10-2-causing-certificate-issues?language=en_US), this is due to constraints in newer versions of openssl that the current certificate files do not meet.
The last time the certificate generation script was really updated (apart from linting) appears to be 2021 (by openQA author @bmwiedemann ...small world!), and that was just to extend some validity periods. The last time it was really overhauled was f242303fb920b1efa3788dc42b0b3380bfe1e3da in 2018. By comparison, the script it was based on - [make_ssl_certs.py from cpython](https://github.com/python/cpython/blob/a0dce37895947a09f3ff97ae33bba703f6a6310c/Lib/test/certdata/make_ssl_certs.py) - has been updated several times to adapt to stricter constraints, including adding the `basicConstraints = critical,CA:false` directive we need here.
Unfortunately the two scripts diverge enough that applying the updates to distributed's version isn't straightforward. I intend to look at it tomorrow, but am filing this in case someone else wants to get to it first, or I don't manage to do it.
**Minimal Complete Verifiable Example**:
Run the test suite with openssl 3.x, I think that should be enough. You'll get failures like:
```
________________ TestClientSecurityLoader.test_security_loader _________________
self =
address = '0.0.0.0:38217', deserialize = True
connection_args = {'extra_conn_args': {}, 'require_encryption': True, 'ssl_context': }
ip = '0.0.0.0', port = 38217
kwargs = {'ssl_options': }
async def connect(self, address, deserialize=True, **connection_args):
self._check_encryption(address, connection_args)
ip, port = parse_host_port(address)
kwargs = self._get_connect_args(**connection_args)
try:
# server_hostname option (for SNI) only works with tornado.iostream.IOStream
if "server_hostname" in kwargs:
stream = await self.client.connect(
ip, port, max_buffer_size=MAX_BUFFER_SIZE
)
stream = await stream.start_tls(False, **kwargs)
else:
> stream = await self.client.connect(
ip, port, max_buffer_size=MAX_BUFFER_SIZE, **kwargs
)
../../BUILDROOT/usr/lib/python3.13/site-packages/distributed/comm/tcp.py:546:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib64/python3.13/site-packages/tornado/tcpclient.py:292: in connect
stream = await stream.start_tls(
/usr/lib64/python3.13/site-packages/tornado/iostream.py:1367: in _do_ssl_handshake
self.socket.do_handshake()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = , block = False
@_sslcopydoc
def do_handshake(self, block=False):
self._check_connected()
timeout = self.gettimeout()
try:
if timeout == 0.0 and block:
self.settimeout(None)
> self._sslobj.do_handshake()
E ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Basic Constraints of CA cert not marked critical (_ssl.c:1020)
/usr/lib64/python3.13/ssl.py:1363: SSLCertVerificationError
```
**Anything else we need to know?**:
**Environment**:
- Dask version: 2024.6.0
- Python version: 3.13
- Operating System: Fedora Rawhide
- Install method (conda, pip, source): distro packages (I am a distro packager working on rebuilds for Python 3.13, including getting distributed packaged, which means getting the tests to pass)
Contributor guide
Assessment
This issue has not been assessed yet.