VLESS REALITY Vision fails on Debian 13 (OpenSSL 3.5.5) — TLS EOF on all clients

VLESS REALITY Vision fails on Debian 13 (Trixie) with OpenSSL 3.5.5 — all TLS clients receive EOF

Environment

• OS: Debian 13 (Trixie), kernel 6.12.74+deb13+1-amd64
• OpenSSL: 3.5.5 (system default)
• curl: 8.14.1 (linked against OpenSSL 3.5.5)
• Xray client: v26.2.6 (also tested with v26.3.27, same result)
• Xray server: v26.2.6 (via 3X-UI v2.8.11)
• Protocol: VLESS + TCP + REALITY + xtls-rprx-vision
• Server SNI/Target: www.microsoft.com

Problem

VLESS REALITY with xtls-rprx-vision flow works on Windows (v2rayN 7.20.2) and mobile clients with the same node configuration, but fails on Debian 13 Trixie. Every TLS client on the system (curl, Python ssl module, Firefox) receives an immediate EOF during the TLS handshake with the destination server (e.g. google.com) after traffic is tunneled through Xray.

Xray log (debug)

Xray successfully establishes the tunnel — no errors on the proxy side:

proxy/vless/outbound: tunneling request to tcp:www.google.com:443 via [server]:443
proxy: XtlsFilterTls found tls client hello! 1574
proxy: XtlsPadding 1574 204 0

No connection reset, no decode errors, no warnings. The tunnel appears fully functional.

Client-side errors

curl (OpenSSL 3.5.5):

$ curl -x socks5h://127.0.0.1:10809 -I https://www.google.com
curl: (35) TLS connect error: error:0A000126:SSL routines::unexpected eof while reading

With IgnoreUnexpectedEOF in openssl.cnf, the error changes to:

curl: (35) TLS connect error: error:00000000:lib(0)::reason(0)

curl verbose output:

* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLS connect error: error:00000000:lib(0)::reason(0)
* OpenSSL SSL_connect: SSL_ERROR_ZERO_RETURN in connection to www.google.com:443

Python 3.13 (ssl module, also OpenSSL 3.5.5):

ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1029)

Firefox (NSS, not OpenSSL):

PR_END_OF_FILE_ERROR

What was tested and ruled out

1. DNS pollution: Confirmed and resolved by using socks5h:// (remote DNS resolution)
2. OpenSSL 3.5 IgnoreUnexpectedEOF: Applied to /etc/ssl/openssl.cnf — changes the error code but does not fix the connection
3. Xray version mismatch: Downgraded client from 26.3.27 to 26.2.6 (matching server) — same result
4. With and without flow: Tested both "flow": "xtls-rprx-vision" and no flow — same result
5. Different fingerprints: Tested chrome, firefox — same result
6. curl -k (skip TLS verify): Same error
7. OPENSSL_CONF=/dev/null: Still fails (rules out openssl.cnf issues)
8. HTTP proxy inbound (port 10810): Same result
9. Multiple nodes: Tested 2 different VPS servers, both fail identically
10. System time: Verified correct
11. Firewall: nft and iptables rules flushed
12. Same node on Windows: Works perfectly with v2rayN 7.20.2
13. Same node on mobile: Works perfectly
14. Debian proxied through Windows TUN: Works — confirming the node itself is functional

Key observation

When Debian's traffic is routed through Windows v2rayN's TUN mode (network-layer proxy), everything works. The failure only occurs when Debian's own TLS stack (OpenSSL 3.5 or Firefox NSS) tries to perform the TLS handshake on data that has passed through XTLS Vision's pipeline.

This suggests XTLS Vision's traffic shaping/padding may produce TLS records that OpenSSL 3.5.5 and recent NSS versions reject as malformed, while older OpenSSL versions (on Windows/mobile) accept.

Minimal reproduction config

{
  "log": {"loglevel": "debug"},
  "dns": {"servers": ["8.8.8.8"]},
  "inbounds": [{
    "port": 10809,
    "protocol": "socks",
    "settings": {"udp": true}
  }],
  "outbounds": [{
    "protocol": "vless",
    "settings": {
      "vnext": [{
        "address": "SERVER_IP",
        "port": 443,
        "users": [{
          "id": "UUID",
          "flow": "xtls-rprx-vision",
          "encryption": "none"
        }]
      }]
    },
    "streamSettings": {
      "network": "tcp",
      "security": "reality",
      "realitySettings": {
        "serverName": "www.microsoft.com",
        "fingerprint": "chrome",
        "publicKey": "PUBLIC_KEY",
        "shortId": "SHORT_ID",
        "spiderX": "/"
      }
    }
  }]
}

Expected behavior

curl/Firefox should complete TLS handshake with the destination server through the Xray VLESS REALITY Vision tunnel, as it does on Windows and mobile platforms.

Actual behavior

All TLS clients on Debian 13 receive an immediate EOF (zero bytes read) during the TLS handshake, despite Xray logs showing a successful tunnel with no errors.

Start by reproducing the Debian 13 failure with the minimal VLESS REALITY Vision configuration and debug logging, then compare the TLS handshake behavior with the working Windows and mobile cases. The issue is resolved when curl, Python, and Firefox complete the destination TLS handshake through the tunnel without EOF errors.