test_ssl often fails against OpenSSL 3.4.0

Bug report

Bug description:

test_ssl often fails on the main, 3.13 and 3.12 branches when running make test. (This is on an M1 MacBook Pro running MacOS 15.1.) Sometimes it fails the first time, succeeds when rerun, fails both times, or doesn't fail at all. Though the tracebacks seem to be substantially different, the fundamental problem on 3.13 and main seems to be a bad TLS certificate. I see a different error on 3.12.

Here's the latest (long) failure traceback on main:

Traceback (most recent call last):
   File "/Users/skip/src/python/cpython/Lib/test/test_ssl.py", line 2262, in wrap_conn
    self.sslconn = self.server.context.wrap_socket(
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.sock, server_side=True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/Users/skip/src/python/cpython/Lib/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sock=sock,
        ^^^^^^^^^^
    ...<5 lines>...
        session=session
        ^^^^^^^^^^^^^^^
    )
    ^
   File "/Users/skip/src/python/cpython/Lib/ssl.py", line 1076, in _create
    self.do_handshake()
    ~~~~~~~~~~~~~~~~~^^
   File "/Users/skip/src/python/cpython/Lib/ssl.py", line 1372, in do_handshake
    self._sslobj.do_handshake()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
 ssl.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate (_ssl.c:1024)
 server:  new connection from ('127.0.0.1', 62987)
 server:  new connection from ('127.0.0.1', 62988)

 server:  bad connection attempt from ('127.0.0.1', 62984):
Traceback (most recent call last):
   File "/Users/skip/src/python/cpython/Lib/test/test_ssl.py", line 2262, in wrap_conn
    self.sslconn = self.server.context.wrap_socket(
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.sock, server_side=True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/Users/skip/src/python/cpython/Lib/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sock=sock,
        ^^^^^^^^^^
    ...<5 lines>...
        session=session
        ^^^^^^^^^^^^^^^
    )
    ^
   File "/Users/skip/src/python/cpython/Lib/ssl.py", line 1076, in _create
    self.do_handshake()
    ~~~~~~~~~~~~~~~~~^^
   File "/Users/skip/src/python/cpython/Lib/ssl.py", line 1372, in do_handshake
    self._sslobj.do_handshake()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
 ssl.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate (_ssl.c:1024)

 server:  bad connection attempt from ('127.0.0.1', 62987):
Traceback (most recent call last):
   File "/Users/skip/src/python/cpython/Lib/test/test_ssl.py", line 2262, in wrap_conn
    self.sslconn = self.server.context.wrap_socket(
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.sock, server_side=True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/Users/skip/src/python/cpython/Lib/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sock=sock,
        ^^^^^^^^^^
    ...<5 lines>...
        session=session
        ^^^^^^^^^^^^^^^
    )
    ^
   File "/Users/skip/src/python/cpython/Lib/ssl.py", line 1076, in _create
    self.do_handshake()
    ~~~~~~~~~~~~~~~~~^^
   File "/Users/skip/src/python/cpython/Lib/ssl.py", line 1372, in do_handshake
    self._sslobj.do_handshake()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
 ssl.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate (_ssl.c:1024)
 server: connection cipher is now ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)
 Connection reset by peer: ('127.0.0.1', 62988)
 server:  new connection from ('127.0.0.1', 62993)

Here's the latest traceback from the 3.13 branch`:

Re-running test_ssl in verbose mode (matching: test_wrong_cert_tls13)
test_ssl: testing with 'OpenSSL 3.4.0 22 Oct 2024' (3, 4, 0, 0, 0)
          under Mac ('15.1', ('', '', ''), 'arm64')
          HAS_SNI = True
          OP_ALL = 0x80000050
          OP_NO_TLSv1_1 = 0x10000000
test_wrong_cert_tls13 (test.test_ssl.ThreadedTests.test_wrong_cert_tls13) ...  server:  new connection from ('127.0.0.1', 53960)

 server:  bad connection attempt from ('127.0.0.1', 53960):
Traceback (most recent call last):
   File "/Users/skip/src/python/py3.13/Lib/test/test_ssl.py", line 2275, in wrap_conn
    self.sslconn = self.server.context.wrap_socket(
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.sock, server_side=True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/Users/skip/src/python/py3.13/Lib/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sock=sock,
        ^^^^^^^^^^
    ...<5 lines>...
        session=session
        ^^^^^^^^^^^^^^^
    )
    ^
   File "/Users/skip/src/python/py3.13/Lib/ssl.py", line 1076, in _create
    self.do_handshake()
    ~~~~~~~~~~~~~~~~~^^
   File "/Users/skip/src/python/py3.13/Lib/ssl.py", line 1372, in do_handshake
    self._sslobj.do_handshake()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
 ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1020)
FAIL

3.12:

% ./python.exe -m test test_ssl
Using random seed: 2271976119
0:00:00 load avg: 1.25 Run 1 test sequentially
0:00:00 load avg: 1.25 [1/1] test_ssl
test test_ssl failed -- Traceback (most recent call last):
  File "/Users/skip/src/python/py3.12/Lib/test/test_ssl.py", line 4882, in test_preauth_data_to_tls_server
    self.assertIn("before TLS handshake with data", wrap_error.args[1])
AssertionError: 'before TLS handshake with data' not found in '[SSL] record layer failure (_ssl.c:1000)'

Just after the above failure, I ran the same command and it succeeded...

In all three cases, _ssl...so was linked against OpenSSL 3.4.0, for example:

% otool -L Modules/_ssl.cpython-313t-darwin.so
Modules/_ssl.cpython-313t-darwin.so:
	/opt/homebrew/opt/openssl@3/lib/libssl.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/opt/homebrew/opt/openssl@3/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)
(py313t) ~/src/python/py3.13% ls -l /opt/homebrew/opt/openssl@3
lrwxr-xr-x  1 skip  admin  25 Oct 31 14:02 /opt/homebrew/opt/openssl@3 -> ../Cellar/openssl@3/3.4.0

I don't recall if I saw failures against earlier OpenSSL versions. Is 3.4.0 maybe different enough that the _ssl.c code has to be tweaked a bit?

CPython versions tested on:

3.12, 3.13, CPython main branch

Operating systems tested on:

macOS

Start by running ./python.exe -m test test_ssl and inspect the failures in Lib/test/test_ssl.py, especially test_wrong_cert_tls13 and test_preauth_data_to_tls_server. Compare the related _ssl.c behavior across the 3.12, 3.13, and main branches with OpenSSL 3.4.0. Done means the tests are reliable and pass with their expected error messages on macOS.