CMS_decrypt_set1_pkey returns 0 but OpenSSL error queue remains empty (no error code)
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 30.8k
- Forks
- 11.5k
- Avg merge
- 10m
- Merged PRs (30d)
- 1
Description
When calling CMS_decrypt_set1_pkey to associate a private key/certificate with a CMS_ContentInfo, the function returns 0 on failure but leaves the OpenSSL error queue empty (ERR_get_error() returns 0). Consequently, callers cannot determine the reason for the failure (e.g. "no recipient matches") via the normal OpenSSL error APIs.
Reproduction steps:
Compile and run the minimal repro (attached). Steps in the repro:
Generate two independent key pairs and two self‑signed certs (cert1/pkey1, cert2/pkey2).
Encrypt some data with CMS_encrypt for cert1.
Call CMS_decrypt_set1_pkey(cms, pkey2, cert2) where pkey2/cert2 is the wrong pair.
Inspect the return value of CMS_decrypt_set1_pkey (expected 0) and the OpenSSL error queue (ERR_get_error() — expected nonzero error code, but actually returns 0).
Expected behavior:
On failure CMS_decrypt_set1_pkey should populate the OpenSSL error queue with a meaningful error code (for example, CMS_R_NO_MATCHING_RECIPIENT) or otherwise make the reason observable through standard OpenSSL error mechanisms. Callers should be able to get the failure reason via ERR_get_error / ERR_peek_last.
Actual behavior:
CMS_decrypt_set1_pkey returns 0, but ERR_get_error() returns 0 — the error queue is empty. No diagnostic information about the failure is available.
Impact:
Callers cannot distinguish the “no matching recipient” case from other failures, leading to difficulty in proper error handling and diagnostics.
Applications using CMS cannot reliably detect or report the specific reason for the failure.
Can lead to incorrect fallback behavior or obscure failures in production.
P.S.
In openssl 1.1.1 the behavior was expected, but was fixed here https://github.com/openssl/openssl/pull/19222#issuecomment-1282652181
Code example:
#include <stdio.h>
#include <stdlib.h>
#include <openssl/evp.h>
#include <openssl/x509.h>
#include <openssl/rsa.h>
#include <openssl/pem.h>
#include <openssl/cms.h>
#include <openssl/err.h>
#include <openssl/bn.h>
static void die_on_err(const char *msg) {
fprintf(stderr, "%s\n", msg);
ERR_print_errors_fp(stderr);
exit(1);
}
static EVP_PKEY *gen_rsa(void) {
EVP_PKEY *pkey = NULL;
RSA *rsa = NULL;
BIGNUM *e = NULL;
pkey = EVP_PKEY_new();
if (!pkey) return NULL;
rsa = RSA_new();
e = BN_new();
if (!rsa || !e) goto err;
if (!BN_set_word(e, RSA_F4)) goto err;
if (!RSA_generate_key_ex(rsa, 2048, e, NULL)) goto err;
if (!EVP_PKEY_assign_RSA(pkey, rsa)) goto err;
/* rsa now owned by pkey */
BN_free(e);
return pkey;
err:
BN_free(e);
RSA_free(rsa);
EVP_PKEY_free(pkey);
return NULL;
}
static X509 *make_selfsigned(EVP_PKEY *pkey, const char *cn) {
X509 *x = X509_new();
if (!x) return NULL;
ASN1_INTEGER_set(X509_get_serialNumber(x), 1);
X509_gmtime_adj(X509_get_notBefore(x), 0);
X509_gmtime_adj(X509_get_notAfter(x), 31536000L);
X509_set_pubkey(x, pkey);
X509_NAME *name = X509_get_subject_name(x);
X509_NAME_add_entry_by_txt(name, "CN", MBSTRING_ASC,
(const unsigned char*)cn, -1, -1, 0);
X509_set_issuer_name(x, name);
if (!X509_sign(x, pkey, EVP_sha256())) { X509_free(x); return NULL; }
return x;
}
int main(void) {
OpenSSL_add_all_algorithms();
ERR_load_crypto_strings();
/* 1) Generate two independent keypairs and self-signed certs */
EVP_PKEY *pkey1 = gen_rsa();
EVP_PKEY *pkey2 = gen_rsa();
if (!pkey1 || !pkey2) die_on_err("keygen failed");
X509 *cert1 = make_selfsigned(pkey1, "Cert1");
X509 *cert2 = make_selfsigned(pkey2, "Cert2");
if (!cert1 || !cert2) die_on_err("cert creation failed");
/* 2) plaintext */
unsigned char data[] = { 0x01, 0x02, 0x03 };
BIO *in = BIO_new_mem_buf(data, sizeof(data));
if (!in) die_on_err("BIO_new_mem_buf failed");
/* 3) encrypt for cert1 */
STACK_OF(X509) *recips = sk_X509_new_null();
sk_X509_push(recips, cert1); /* do not free cert1 via stack; we'll free cert1 later */
CMS_ContentInfo *cms = CMS_encrypt(recips, in, EVP_aes_256_cbc(), 0);
sk_X509_free(recips);
BIO_free(in);
if (!cms) die_on_err("CMS_encrypt failed");
/* 4) try to bind wrong key (pkey2) and wrong cert (cert2) */
BIO *out = BIO_new(BIO_s_mem());
if (!out) die_on_err("BIO_new failed");
/* CMS_decrypt_set1_pkey returns 0 on failure, 1 on success */
int ok = CMS_decrypt_set1_pkey(cms, pkey2, cert2);
if (ok) {
/* Unexpected: wrong key accepted */
fprintf(stderr, "UNEXPECTED: decryption accepted wrong key\n");
BIO_free(out);
CMS_ContentInfo_free(cms);
EVP_PKEY_free(pkey1); EVP_PKEY_free(pkey2);
X509_free(cert1); X509_free(cert2);
ERR_free_strings();
return 2;
} else {
/* Expected: failure. Check reason */
unsigned long err = ERR_get_error();
if (ERR_GET_REASON(err) == CMS_R_NO_MATCHING_RECIPIENT) {
printf("OK: no matching recipient (expected)\n");
BIO_free(out);
CMS_ContentInfo_free(cms);
EVP_PKEY_free(pkey1); EVP_PKEY_free(pkey2);
X509_free(cert1); X509_free(cert2);
ERR_free_strings();
return 0;
} else {
fprintf(stderr, "FAIL: unexpected error reason\n");
ERR_print_errors_fp(stderr);
BIO_free(out);
CMS_ContentInfo_free(cms);
EVP_PKEY_free(pkey1); EVP_PKEY_free(pkey2);
X509_free(cert1); X509_free(cert2);
ERR_free_strings();
return 3;
}
}
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is named in the report. Start by running the supplied minimal C++ reproduction and inspect CMS_decrypt_set1_pkey, checking ERR_get_error and ERR_peek_last for the wrong key/certificate case. Done means the failure reason is observable through the standard OpenSSL error APIs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, cpp
- Domain
- cryptography, security
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100