certified key selection not working in all cases

The certified key selection implemented by rustls_client_hello_select_certified_key() currently does not take the limitations of the later defined rustls_connection into account. This leads to handshake failures.

Assuming you have a server with 2 certified keys: P-384 and RSA.
The rustls_server_config has:

• TLS 1.2 and 1.3 enabled
• All 1.3 ciphers and only the RSA 1.2 ciphers

A client connecting with TLS 1.2, the rustls_client_hello_select_certified_key() will chose the P-384 certified key (since the client has the necessary ciphers), create the rustls_connection, start the handshake and fail. The connection simply is unable to do RSA with the P-384 certified key.

The fix would be for rustls_client_hello_select_certified_key() to take the server ciphers into account as well. This is a bit awkward, since the certified key needs to be selected before the rustls_connection for the handshake is created.

At the moment, this is a rare scenario. But selecting the certified key on incomplete understanding of the TLS properties involved, could lead to more common failures in the future. When a new rustls version is released and disables ciphers or particular combinations of them.

Supporting multiple certified keys in a server was helpful in the introduction of EC. The continued support of such a feature via rustls_client_hello_select_certified_key() carries good intentions, however this case shows that it is incomplete.

I guess, either certified key selection is implemented deep inside the rustls handshake itself or it needs to be abandoned.

Opinions?

PS. minor detail: the error of such a failure gives [7107] peer is incompatible: reason omitted. Where does this reason omitted come from? In the rustls code I see only cases of this error with descriptions.

Start by tracing rustls_client_hello_select_certified_key() and the later rustls_connection creation to understand which TLS properties each considers. Reproduce the TLS 1.2 case with P-384 and RSA keys, then define whether selection should account for server ciphers or move into the handshake; also trace the source of the "reason omitted" error. Done requires an agreed design and regression coverage.