openssl / openssl/openssl

TLS1.3 with brainpool version incompatibility

Open
#25,697 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triaged: bug
Dominant language
C
Stars
30.8k
Forks
11.5k
Avg merge
10m
Merged PRs (30d)
1

Description

When an openssl-3.0 client wants to support brainpool server certificates,
but also allow TLS1.3 with RSA certificates, everything works fine, when
both client and server enable brainpool groups, in addition to nist groups.
So the client has to use this command line:
./openssl s_client -groups P-256:brainpoolP256r1
and the openssl-3.0 server uses this command line for brainpool:
./openssl s_server -cert ../test/certs/server-ecdsa-brainpoolP256r1-cert.pem -key ../test/certs/server-ecdsa-brainpoolP256r1-key.pem -groups P-256:brainpoolP256r1
or this command line for RSA:
./openssl s_server
everything works fine, RSA use TLS1.3 and ECC use TLS1.2.
Also if both server and client use openssl-3.2 everything works as well,
except the ECC use TLS1.3, as expected.
BUT the openssl-3.0 client cannot connect to the openssl-3.2 server
when the server uses brainpool:

$ ./openssl s_client -groups P-256:brainpoolP256r1 -trace
CONNECTED(00000003)
Sent Record
Header:
  Version = TLS 1.0 (0x301)
  Content Type = Handshake (22)
  Length = 327
    ClientHello, Length=323
      client_version=0x303 (TLS 1.2)
      Random:
        gmt_unix_time=0xD3B13943
        random_bytes (len=28): 24530A97534115EA57B17C77ABAC33F909733FDBAA73C61AD594C525
      session_id (len=32): 397A7086775ACCDB82A75E2B19B7FE6459BF3088B322A7A5D8EAC17791A76D40
      cipher_suites (len=62)
        {0x13, 0x02} TLS_AES_256_GCM_SHA384
        {0x13, 0x03} TLS_CHACHA20_POLY1305_SHA256
        {0x13, 0x01} TLS_AES_128_GCM_SHA256
        {0xC0, 0x2C} TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        {0xC0, 0x30} TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        {0x00, 0x9F} TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
        {0xCC, 0xA9} TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
        {0xCC, 0xA8} TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
        {0xCC, 0xAA} TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256
        {0xC0, 0x2B} TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        {0xC0, 0x2F} TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        {0x00, 0x9E} TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
        {0xC0, 0x24} TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
        {0xC0, 0x28} TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
        {0x00, 0x6B} TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        {0xC0, 0x23} TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        {0xC0, 0x27} TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
        {0x00, 0x67} TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        {0xC0, 0x0A} TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
        {0xC0, 0x14} TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        {0x00, 0x39} TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        {0xC0, 0x09} TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
        {0xC0, 0x13} TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        {0x00, 0x33} TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        {0x00, 0x9D} TLS_RSA_WITH_AES_256_GCM_SHA384
        {0x00, 0x9C} TLS_RSA_WITH_AES_128_GCM_SHA256
        {0x00, 0x3D} TLS_RSA_WITH_AES_256_CBC_SHA256
        {0x00, 0x3C} TLS_RSA_WITH_AES_128_CBC_SHA256
        {0x00, 0x35} TLS_RSA_WITH_AES_256_CBC_SHA
        {0x00, 0x2F} TLS_RSA_WITH_AES_128_CBC_SHA
        {0x00, 0xFF} TLS_EMPTY_RENEGOTIATION_INFO_SCSV
      compression_methods (len=1)
        No Compression (0x00)
      extensions, length = 188
        extension_type=server_name(0), length=14
          0000 - 00 0c 00 00 09 6c 6f 63-61 6c 68 6f 73 74      .....localhost
        extension_type=ec_point_formats(11), length=4
          uncompressed (0)
          ansiX962_compressed_prime (1)
          ansiX962_compressed_char2 (2)
        extension_type=supported_groups(10), length=6
          secp256r1 (P-256) (23)
          brainpoolP256r1 (26)
        extension_type=session_ticket(35), length=0
        extension_type=encrypt_then_mac(22), length=0
        extension_type=extended_master_secret(23), length=0
        extension_type=signature_algorithms(13), length=42
          ecdsa_secp256r1_sha256 (0x0403)
          ecdsa_secp384r1_sha384 (0x0503)
          ecdsa_secp521r1_sha512 (0x0603)
          ed25519 (0x0807)
          ed448 (0x0808)
          rsa_pss_pss_sha256 (0x0809)
          rsa_pss_pss_sha384 (0x080a)
          rsa_pss_pss_sha512 (0x080b)
          rsa_pss_rsae_sha256 (0x0804)
          rsa_pss_rsae_sha384 (0x0805)
          rsa_pss_rsae_sha512 (0x0806)
          rsa_pkcs1_sha256 (0x0401)
          rsa_pkcs1_sha384 (0x0501)
          rsa_pkcs1_sha512 (0x0601)
          ecdsa_sha224 (0x0303)
          rsa_pkcs1_sha224 (0x0301)
          dsa_sha224 (0x0302)
          dsa_sha256 (0x0402)
          dsa_sha384 (0x0502)
          dsa_sha512 (0x0602)
        extension_type=supported_versions(43), length=9
          TLS 1.3 (772)
          TLS 1.2 (771)
          TLS 1.1 (770)
          TLS 1.0 (769)
        extension_type=psk_key_exchange_modes(45), length=2
          psk_dhe_ke (1)
        extension_type=key_share(51), length=71
            NamedGroup: secp256r1 (P-256) (23)
            key_exchange:  (len=65): 04B9F00C469BB69090E6B7E8FC0EF518A600EEA9950825CFAFBC88DCF2B0EE218100ABD95B8C6CFAA29A7302332E42FCB07286C0653C823E5896981ED97C363D0D

Received Record
Header:
  Version = TLS 1.2 (0x303)
  Content Type = Alert (21)
  Length = 2
    Level=fatal(2), description=handshake failure(40)

40778BA11E7F0000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:ssl/record/rec_layer_s3.c:1605:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 332 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Reason is the openssl-3.2 server chooses TLS1.3 although there is no
signature algorithm ecdsa_brainpoolP256r1_sha256 advertised by the
openssl-3.0 client, so that is bound to fail.
As a workaround either the openssl-3.0 client or the openssl-3.2 server could
disable tls1_3 but that is not desirable for the RSA server.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the shown openssl s_client and s_server commands, using the referenced brainpool certificate and key files, and compare the TLS trace with OpenSSL 3.0 and 3.2. Start by examining the TLS 1.3 signature-algorithm and supported-group negotiation; done means an OpenSSL 3.0 client can connect to the OpenSSL 3.2 brainpool server while TLS 1.3 remains available for RSA.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
cryptography, networking, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.