Lack of automatic key update

OpenSSL's TLS stack does not perform automatic key update (TLS 1.3) or renegotiation (TLS 1.2 and earlier) when an AEAD limit is approached. Moreover, we do not do any kind of AEAD usage counting in libssl, so when an AEAD limit is reached, we will just continue operating.

This means that if a large amount of data is transferred over a single connection, the AEAD limit will be exceeded and the confidentiality and integrity assurances an AEAD is intended to provide will be impaired.

There is a possibility that this issue is mitigated in FIPS mode due to an EVP_CIPHER_CTX enforcing usage limits in FIPS mode but this would depend on whether we are reusing the same EVP_CIPHER_CTX in libssl, and would need to be confirmed and verified experimentally. In any case this does not apply to non-FIPS usage.

The fix would be as follows:

TLS 1.3

For TLS 1.3 the fix is simple:

• Add an AEAD usage counter.
• Add information on the AEAD limits for each ciphersuite.
• Trigger key update automatically well before the AEAD limit is reached.
• If we somehow do reach the AEAD limit (or get near it) and the automatic key update did not occur for some reason, fail the connection.
• Add tests for all this logic (update and backstop enforcement) based on an artificially low AEAD limit

TLS 1.2

For TLS 1.2 and earlier things are more complicated. The only means we have to cycle the AEAD keys is renegotiation. This is more "application-visible":

• Renegotiating might cause a peer to choose different parameters, or certificates, or so on, and this may result in application visible side effects.
• To my knowledge after a number of vulnerabilities caused by improper implementation of renegotiation, it has become fairly common for people to disable renegotiation support in software configurations.
• In this circumstance to my knowledge we have no means of cycling the AEAD keys, so the only option left here is to terminate the connection if we approach an AEAD limit.

The question is mainly one of compatibility. Regardless of what we choose for TLS 1.2, we should not allow a connection to continue being used if the AEAD limit is reached. The question is if we want to start doing automatic renegotiation if a peer has it enabled. So the options are:

1. Ship a patch release which does automatic renegotiation if available, and fails hard if the AEAD limit is reached (+1)
2. Ship a patch release which fails hard if the AEAD limit is reached, and a subsequent feature release which does automatic renegotiation by default (also +1)
3. Ship a patch release which fails hard if the AEAD limit is reached, but keep automatic renegotiation opt-in (not terribly nice)

1 and 2 could have an API to disable it (default on), if it is deemed really necessary.

OTC has agreed that this does not need to be handled as a CVE, thus moving this to a public issue.

Start in libssl by tracing AEAD usage and the existing TLS 1.3 key-update and TLS 1.2 renegotiation paths; verify experimentally whether reused EVP_CIPHER_CTX instances enforce limits in FIPS mode. Define artificially low AEAD-limit tests for automatic updates and connection termination, then resolve the compatibility and opt-in questions for TLS 1.2 before implementation.