RFC 5246 violation: OpenSSL TLS 1.2 client silently accepts a zero-byte Handshake record

Version

openssl-3.6.3

Description

An OpenSSL TLS 1.2 client silently accepts a TLS record of ContentType
handshake (0x16) carrying a zero-byte payload (16 03 03 00 00), without sending any
alert and without aborting the connection. The client continues the handshake normally and
eventually completes it with both sides exchanging Finished messages.

The zero-byte record is injected between the server's Certificate and ServerKeyExchange
messages, while the client is in the state waiting for the ServerKeyExchange.

Trying to inject such a TLS record in TLS 1.3 result in the parties rejecting the message and closing the connection

Impact

RFC violation. RFC 5246 §7.4 defines the Handshake struct as requiring at minimum a
4-byte header (1 byte msg_type + 3 bytes length); a zero-byte payload cannot encode
any valid handshake message. RFC 5246 §7.2.2 lists decode_error as the mandatory fatal
alert for messages whose length is incorrect. By accepting a zero-byte Handshake record
without error, OpenSSL silently swallows data that no conforming implementation should
accept.

Acceptance of a zero-byte Handshake record

Reproduction steps

The following Python script acts as a transparent MITM proxy. It forwards all TLS 1.2
messages from an upstream OpenSSL server to the OpenSSL client unchanged, except that it
injects a single zero-byte Handshake record immediately after the Certificate message.

Generate a self-signed RSA test certificate if needed:

openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem \
        -days 365 -nodes -subj '/CN=test'

Start an OpenSSL 3.4.0 TLS 1.2 server on port 4434:

openssl s_server -accept 4434 -cert cert.pem -key key.pem -tls1_2 -no_ticket -quiet

Run the proxy on port 4433:

#!/usr/bin/env python3
"""
MITM proxy — injects 16 03 03 00 00 (zero-byte Handshake record) between
the server Certificate and ServerKeyExchange messages.

Run:
  (terminal 1) openssl s_server -accept 4434 -cert cert.pem -key key.pem \\
                                -tls1_2 -no_ticket -quiet
  (terminal 2) python3 mitm.py
  (terminal 3) openssl s_client -connect localhost:4433 -tls1_2 \\
                                -no_ticket -quiet < /dev/null
"""
import socket
import struct

PROXY_PORT    = 4433
UPSTREAM_PORT = 4434
ZERO_HS       = b"\x16\x03\x03\x00\x00"


def read_exact(s, n):
    buf = b""
    while len(buf) < n:
        d = s.recv(n - len(buf))
        if not d:
            raise EOFError
        buf += d
    return buf


def recv_record(s):
    h = read_exact(s, 5)
    b = read_exact(s, struct.unpack(">H", h[3:5])[0])
    return h[0], h, b


def split_hs(body):
    """Yield (hs_type, data) for each handshake message in a Handshake record body."""
    i = 0
    while i + 4 <= len(body):
        ht = body[i]
        hl = struct.unpack(">I", b"\x00" + body[i + 1:i + 4])[0]
        yield ht, body[i + 4:i + 4 + hl]
        i += 4 + hl


def make_hs_record(ht, data):
    inner = bytes([ht]) + struct.pack(">I", len(data))[1:] + data
    return b"\x16\x03\x03" + struct.pack(">H", len(inner)) + inner


HS_NAMES = {
    2: "ServerHello", 11: "Certificate",
    12: "ServerKeyExchange", 14: "ServerHelloDone",
}
ALERT_NAMES = {
    10: "unexpected_message", 40: "handshake_failure", 50: "decode_error",
}

srv = socket.socket()
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("", PROXY_PORT))
srv.listen(1)
print(f"Proxy :{PROXY_PORT} → :{UPSTREAM_PORT}\n")

cli, _ = srv.accept()
up = socket.socket()
up.connect(("127.0.0.1", UPSTREAM_PORT))

injected = alert_seen = False
finished = 0

try:
    # Forward ClientHello
    ct, h, b = recv_record(cli)
    up.sendall(h + b)
    print("C→S  ClientHello")

    # Forward server flight message by message; inject after Certificate
    shd_done = False
    while not shd_done:
        ct, h, b = recv_record(up)
        if ct == 0x16:
            for ht, hd in split_hs(b):
                print(f"S→C  {HS_NAMES.get(ht, f'Handshake({ht})')}")
                cli.sendall(make_hs_record(ht, hd))
                if ht == 11 and not injected:
                    print("***  INJECT 16 03 03 00 00  (zero-byte Handshake record)")
                    cli.sendall(ZERO_HS)
                    injected = True
                if ht == 14:
                    shd_done = True
        else:
            cli.sendall(h + b)

    # Client sends ClientKeyExchange + ChangeCipherSpec + Finished(enc)
    # In TLS 1.2, the encrypted Finished has ContentType=Handshake (0x16).
    cli.settimeout(3.0)
    up.settimeout(3.0)
    client_ccs = False
    for _ in range(5):
        try:
            ct, h, b = recv_record(cli)
        except (socket.timeout, EOFError):
            break
        up.sendall(h + b)
        if ct == 0x15:
            lvl = "fatal" if b[0] == 2 else "warning"
            desc = ALERT_NAMES.get(b[1], b[1])
            print(f"C→S  Alert({lvl}, {desc})")
            alert_seen = True
            break
        elif ct == 0x14:
            print("C→S  ChangeCipherSpec")
            client_ccs = True
        elif ct == 0x16 and client_ccs:
            print("C→S  Finished(enc)")
            finished += 1
            break

    # Server sends ChangeCipherSpec + Finished(enc)
    if not alert_seen and finished >= 1:
        server_ccs = False
        for _ in range(5):
            try:
                ct, h, b = recv_record(up)
            except (socket.timeout, EOFError):
                break
            cli.sendall(h + b)
            if ct == 0x14:
                print("S→C  ChangeCipherSpec")
                server_ccs = True
            elif ct == 0x16 and server_ccs:
                print("S→C  Finished(enc)")
                finished += 1
                break

except (EOFError, OSError):
    pass
finally:
    cli.close()
    up.close()
    srv.close()

print()
if alert_seen:
    print("[+] CORRECT: client aborted with Alert — RFC 5246 §7.4 respected.")
elif finished >= 2:
    print("[!] BUG confirmed: handshake completed despite injected zero-byte Handshake "
          "record.\n    OpenSSL accepted the record silently, violating RFC 5246 §7.4.")
else:
    print("[-] Inconclusive.")

Connect an OpenSSL 3.4.0 client through the proxy:

openssl s_client -connect localhost:4433 -tls1_2 -no_ticket -quiet < /dev/null

Expected behavior (RFC 5246 §7.4 + §7.2.2): the client MUST send an alert and abort the connection upon receiving a Handshake record whose payload is too short to contain any valid handshake message.

Acknowledgements

This bug was found thanks to the tlspuffin fuzzer
designed and developed by the tlspuffin team:

• Nataël Baffou — Engineer, Inria, France
• Olivier Demengeon — Engineer, Inria, France
• Tom Gouville — PhD student, Inria, France
• Lucca Hirschi — Researcher, Inria, France
• Steve Kremer — Researcher, Inria, France

Reproduce the TLS 1.2 case with openssl s_server, openssl s_client, and the provided Python MITM proxy, then trace OpenSSL's record and handshake processing for the injected zero-byte Handshake record. Done means the client rejects the record with a fatal decode_error alert and aborts instead of completing the handshake; verify the behavior against the TLS 1.3 comparison in the report.