Error in mTLS in node js using PKCS11 proxy and HSM

Description

I am implementing mutual TLS (mTLS) between a Node.js server and client using private keys stored in an HSM, accessed via a local PKCS#11 proxy on each machine.

Setup

• Both server and client have their own HSM + PKCS#11 proxy running locally
• Server and client are running on different machines
• Each side uses its own HSM instance for private key operations
• Private keys are accessed via PKCS#11 proxy
• Certificates are loaded from local PEM files

Architecture Diagram

        ┌──────────────────────────────┐          ┌──────────────────────────────┐
        │        Server Machine        │          │        Client Machine        │
        │                              │          │                              │
        │   +----------------------+   │          │   +----------------------+   │
        │   |        HSM           |   │          │   |        HSM           |   │
        │   |  (Private Key:      |   │          │   |  (Private Key:      |   │
        │   |   CORE0001)         |   │          │   |   EDGE0001)         |   │
        │   +----------+-----------+  │          │   +----------+-----------+  │
        │              |              │          │              |              │
        │              | PKCS#11      │          │              | PKCS#11      │
        │              |              │          │              |              │
        │   +----------v-----------+  │          │   +----------v-----------+  │
        │   |   PKCS#11 Proxy      |  │          │   |   PKCS#11 Proxy      |  │
        │   |   (Local)            |  │          │   |   (Local)            |  │
        │   +----------+-----------+  │          │   +----------+-----------+  │
        │              |              │          │              |              │
        │   +----------v-----------+  │          │   +----------v-----------+  │
        │   |    Node.js Server    |  │          │   |    Node.js Client    |  │
        │   |    (socket.io)       |  │          │   | (socket.io-client)   |  │
        │   +----------------------+  │          │   +----------------------+  │
        │                              │          │                              │
        └──────────────┬───────────────┘          └──────────────┬───────────────┘
                       │                                         │
                       │<---------- mTLS (HTTPS) --------------->│
                       │                                         │

Libraries

• Server: socket.io (^4.8.0)
• Client: socket.io-client (^4.8.0)

Server (Working)

Key TLS config:

const httpsOptions = {
  cert: serverCert,
  privateKeyEngine: "pkcs11",
  privateKeyIdentifier: PKCS11_KEY_URI,
  ca: caCert,
  requestCert: true,
  rejectUnauthorized: true,
  minVersion: "TLSv1.2",
  maxVersion: "TLSv1.2",
};

Server starts successfully and loads the private key from HSM without issues.

Client (Failing)

Key TLS setup:

const secureContext = createSecureContext({
  cert: clientCert,
  privateKeyEngine: "pkcs11",
  privateKeyIdentifier: PKCS11_KEY_URI,
  ca: caCert,
  minVersion: "TLSv1.2",
});

const agent = new HttpsAgent({
  secureContext,
  rejectUnauthorized: true,
  checkServerIdentity: () => undefined,
});

Socket.IO connection:

const socket = io(SERVER_URL, {
  agent,
  rejectUnauthorized: true,
  transports: ["websocket", "polling"],
  timeout: 20000,
});

Issue

• Client connection times out
• No TLS handshake is completed
• No useful TLS errors on client or server
• Server never logs secureConnection

Observations

• Server works correctly with PKCS#11 key
• Client fails only when using PKCS#11 private key
• Both sides use same CA
• Network connectivity is confirmed
• Each side uses its own HSM + PKCS#11 proxy locally

Expected Behavior

Client should complete TLS handshake using private key from HSM and connect successfully.

Question

Is there any limitation or required configuration when using privateKeyEngine + privateKeyIdentifier with socket.io-client (via https.Agent)?

Any guidance on debugging PKCS#11-based TLS handshake in Node.js would be helpful.

No repository file or test is named. Start at the socket.io-client connection path using the Node.js createSecureContext and https.Agent configuration, then reproduce the client handshake with TLS diagnostics enabled. Done means identifying whether the PKCS#11 options are supported there and documenting a confirmed configuration or limitation.