openssl / openssl/openssl

CMS_decrypt_set1_pkey returns 0 but OpenSSL error queue remains empty (no error code)

Open
#31,632 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted triaged: feature
Dominant language
C
Stars
30.8k
Forks
11.5k
Avg merge
10m
Merged PRs (30d)
1

Description

When calling CMS_decrypt_set1_pkey to associate a private key/certificate with a CMS_ContentInfo, the function returns 0 on failure but leaves the OpenSSL error queue empty (ERR_get_error() returns 0). Consequently, callers cannot determine the reason for the failure (e.g. "no recipient matches") via the normal OpenSSL error APIs.

Reproduction steps:

Compile and run the minimal repro (attached). Steps in the repro:
Generate two independent key pairs and two self‑signed certs (cert1/pkey1, cert2/pkey2).
Encrypt some data with CMS_encrypt for cert1.
Call CMS_decrypt_set1_pkey(cms, pkey2, cert2) where pkey2/cert2 is the wrong pair.
Inspect the return value of CMS_decrypt_set1_pkey (expected 0) and the OpenSSL error queue (ERR_get_error() — expected nonzero error code, but actually returns 0).
Expected behavior:
On failure CMS_decrypt_set1_pkey should populate the OpenSSL error queue with a meaningful error code (for example, CMS_R_NO_MATCHING_RECIPIENT) or otherwise make the reason observable through standard OpenSSL error mechanisms. Callers should be able to get the failure reason via ERR_get_error / ERR_peek_last.

Actual behavior:
CMS_decrypt_set1_pkey returns 0, but ERR_get_error() returns 0 — the error queue is empty. No diagnostic information about the failure is available.

Impact:

Callers cannot distinguish the “no matching recipient” case from other failures, leading to difficulty in proper error handling and diagnostics.
Applications using CMS cannot reliably detect or report the specific reason for the failure.
Can lead to incorrect fallback behavior or obscure failures in production.

P.S.
In openssl 1.1.1 the behavior was expected, but was fixed here https://github.com/openssl/openssl/pull/19222#issuecomment-1282652181

Code example:

#include <stdio.h>
#include <stdlib.h>
#include <openssl/evp.h>
#include <openssl/x509.h>
#include <openssl/rsa.h>
#include <openssl/pem.h>
#include <openssl/cms.h>
#include <openssl/err.h>
#include <openssl/bn.h>

static void die_on_err(const char *msg) {
    fprintf(stderr, "%s\n", msg);
    ERR_print_errors_fp(stderr);
    exit(1);
}

static EVP_PKEY *gen_rsa(void) {
    EVP_PKEY *pkey = NULL;
    RSA *rsa = NULL;
    BIGNUM *e = NULL;

    pkey = EVP_PKEY_new();
    if (!pkey) return NULL;

    rsa = RSA_new();
    e = BN_new();
    if (!rsa || !e) goto err;

    if (!BN_set_word(e, RSA_F4)) goto err;
    if (!RSA_generate_key_ex(rsa, 2048, e, NULL)) goto err;

    if (!EVP_PKEY_assign_RSA(pkey, rsa)) goto err;
    /* rsa now owned by pkey */
    BN_free(e);
    return pkey;

err:
    BN_free(e);
    RSA_free(rsa);
    EVP_PKEY_free(pkey);
    return NULL;
}

static X509 *make_selfsigned(EVP_PKEY *pkey, const char *cn) {
    X509 *x = X509_new();
    if (!x) return NULL;

    ASN1_INTEGER_set(X509_get_serialNumber(x), 1);
    X509_gmtime_adj(X509_get_notBefore(x), 0);
    X509_gmtime_adj(X509_get_notAfter(x), 31536000L);
    X509_set_pubkey(x, pkey);

    X509_NAME *name = X509_get_subject_name(x);
    X509_NAME_add_entry_by_txt(name, "CN", MBSTRING_ASC,
                               (const unsigned char*)cn, -1, -1, 0);
    X509_set_issuer_name(x, name);

    if (!X509_sign(x, pkey, EVP_sha256())) { X509_free(x); return NULL; }
    return x;
}

int main(void) {
    OpenSSL_add_all_algorithms();
    ERR_load_crypto_strings();

    /* 1) Generate two independent keypairs and self-signed certs */
    EVP_PKEY *pkey1 = gen_rsa();
    EVP_PKEY *pkey2 = gen_rsa();
    if (!pkey1 || !pkey2) die_on_err("keygen failed");

    X509 *cert1 = make_selfsigned(pkey1, "Cert1");
    X509 *cert2 = make_selfsigned(pkey2, "Cert2");
    if (!cert1 || !cert2) die_on_err("cert creation failed");

    /* 2) plaintext */
    unsigned char data[] = { 0x01, 0x02, 0x03 };
    BIO *in = BIO_new_mem_buf(data, sizeof(data));
    if (!in) die_on_err("BIO_new_mem_buf failed");

    /* 3) encrypt for cert1 */
    STACK_OF(X509) *recips = sk_X509_new_null();
    sk_X509_push(recips, cert1); /* do not free cert1 via stack; we'll free cert1 later */
    CMS_ContentInfo *cms = CMS_encrypt(recips, in, EVP_aes_256_cbc(), 0);
    sk_X509_free(recips);
    BIO_free(in);
    if (!cms) die_on_err("CMS_encrypt failed");

    /* 4) try to bind wrong key (pkey2) and wrong cert (cert2) */
    BIO *out = BIO_new(BIO_s_mem());
    if (!out) die_on_err("BIO_new failed");

    /* CMS_decrypt_set1_pkey returns 0 on failure, 1 on success */
    int ok = CMS_decrypt_set1_pkey(cms, pkey2, cert2);
    if (ok) {
        /* Unexpected: wrong key accepted */
        fprintf(stderr, "UNEXPECTED: decryption accepted wrong key\n");
        BIO_free(out);
        CMS_ContentInfo_free(cms);
        EVP_PKEY_free(pkey1); EVP_PKEY_free(pkey2);
        X509_free(cert1); X509_free(cert2);
        ERR_free_strings();
        return 2;
    } else {
        /* Expected: failure. Check reason */
        unsigned long err = ERR_get_error();
        if (ERR_GET_REASON(err) == CMS_R_NO_MATCHING_RECIPIENT) {
            printf("OK: no matching recipient (expected)\n");
            BIO_free(out);
            CMS_ContentInfo_free(cms);
            EVP_PKEY_free(pkey1); EVP_PKEY_free(pkey2);
            X509_free(cert1); X509_free(cert2);
            ERR_free_strings();
            return 0;
        } else {
            fprintf(stderr, "FAIL: unexpected error reason\n");
            ERR_print_errors_fp(stderr);
            BIO_free(out);
            CMS_ContentInfo_free(cms);
            EVP_PKEY_free(pkey1); EVP_PKEY_free(pkey2);
            X509_free(cert1); X509_free(cert2);
            ERR_free_strings();
            return 3;
        }
    }
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named in the report. Start by running the supplied minimal C++ reproduction and inspect CMS_decrypt_set1_pkey, checking ERR_get_error and ERR_peek_last for the wrong key/certificate case. Done means the failure reason is observable through the standard OpenSSL error APIs.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, cpp
Domain
cryptography, security
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.