x509/verify tools spin CPU high for unexpected processing of PKCS12 files

The "x509" tool also accepts PKCS#12 files and processes any complex PBE-MAC without further ado. This undocumented and unexpected behavior could allow attackers to disrupt user workflows based on automatic usage of the x509 tool. The attached file is a simple PKCS12 with an empty password and an iteration count of 2147483647.

 openssl x509 -in mac_sha_iter_maxint.p12

mac_sha_iter_maxint.zip

The result is that the x509 spins the CPU to 100% for as long as the iteration needs. For the attached file about half an hour was measured with the time utility.

Similar result with the "verify" tool.

Here is the corresponding stack trace in case of the x509 tool, as in current 3.0.x on Ubuntu 22.04:

(gdb) bt
#0 sha256_block_data_order_avx () at crypto/sha/sha256-x86_64.s:3917
#1 0x00007fa59f12a004 in SHA256_Final (
md=0x55b5053b8b40 "\377\353\335\373S\341Ҹ\333\301Z\310\017\364\254\246\212*\314PW\232\177\250\365\226\306\352\226\357", <incomplete sequence \374>,
c=0x55b5053f1070) at ../include/crypto/md32_common.h:215
#2 0x00007fa59f180957 in sha256_internal_final (outsz=, outl=0x7fffd212b6a0, out=, ctx=)
at ../providers/implementations/digests/sha2_prov.c:72
#3 sha256_internal_final (ctx=, out=, outl=0x7fffd212b6a0, outsz=)
at ../providers/implementations/digests/sha2_prov.c:72
#4 0x00007fa59f0a4b86 in EVP_DigestFinal_ex (ctx=0x55b5053f0ef0,
md=0x55b5053b8b40 "\377\353\335\373S\341Ҹ\333\301Z\310\017\364\254\246\212*\314PW\232\177\250\365\226\306\352\226\357", <incomplete sequence \374>,
isize=0x0) at ../crypto/evp/digest.c:446
#5 0x00007fa59f195f8f in pkcs12kdf_derive (n=32, out=0x7fffd212ba90 "E", md_type=0x55b5053d9dc0, iter=2147483647, id=3, saltlen=,
salt=, passlen=, pass=) at ../providers/implementations/kdfs/pkcs12kdf.c:103
#6 kdf_pkcs12_derive (params=, keylen=, key=, vctx=)
at ../providers/implementations/kdfs/pkcs12kdf.c:220
#7 kdf_pkcs12_derive (vctx=, key=, keylen=, params=)
at ../providers/implementations/kdfs/pkcs12kdf.c:200
#8 0x00007fa59f103bb7 in PKCS12_key_gen_uni_ex (pass=0x55b5053e6460 "", passlen=2, salt=salt@entry=0x55b5053b8a90 "\026", <incomplete sequence \350>,
saltlen=saltlen@entry=8, id=, id@entry=3, iter=, iter@entry=2147483647, n=32, out=0x7fffd212ba90 "E", md_type=0x55b5053d9dc0,
libctx=0x0, propq=0x0) at ../crypto/pkcs12/p12_key.c:121
#9 0x00007fa59f1053bc in PKCS12_key_gen_utf8_ex (pass=, passlen=, salt=0x55b5053b8a90 "\026", <incomplete sequence \350>,
saltlen=8, id=3, iter=2147483647, n=32, out=0x7fffd212ba90 "E", md_type=0x55b5053d9dc0, ctx=0x0, propq=0x0) at ../crypto/pkcs12/p12_key.c:65
#10 0x00007fa59f1b9b16 in pkcs12_gen_mac.constprop.0 (p12=p12@entry=0x55b5053b8b20, pass=0x7fa59f26dd3c "", passlen=0, mac=mac@entry=0x7fffd212bb90 "",
maclen=maclen@entry=0x7fffd212bb84, pkcs12_key_gen=) at ../crypto/pkcs12/p12_mutl.c:143
#11 0x00007fa59f1040fe in PKCS12_verify_mac (p12=0x55b5053b8b20, pass=, passlen=) at ../crypto/pkcs12/p12_mutl.c:185
#12 0x00007fa59f1b9466 in try_pkcs12.constprop.0 (data=0x7fffd212c140, v=0x7fffd212c630, ctx=0x55b5053a7ca0, propq=, libctx=)
at ../crypto/store/store_result.c:538
#13 0x00007fa59f133496 in ossl_store_handle_load_result (params=, arg=0x7fffd212c630) at ../crypto/store/store_result.c:146
#14 0x00007fa59f090693 in decoder_process (params=params@entry=0x7fffd212c2e0, arg=arg@entry=0x7fffd212c440) at ../crypto/encode_decode/decoder_lib.c:737
#15 0x00007fa59f1b8939 in any2obj_decode_final.constprop.0 (objtype=, objtype@entry=0, mem=0x55b5053b78c0,
data_cb=data_cb@entry=0x7fa59f0905f0 <decoder_process>, data_cbarg=data_cbarg@entry=0x7fffd212c440, provctx=)
at ../providers/implementations/storemgmt/file_store_any2obj.c:76
#16 0x00007fa59f1b126f in der2obj_decode (provctx=, cin=, selection=, data_cb=0x7fa59f0905f0 <decoder_process>,
data_cbarg=0x7fffd212c440, pw_cb=, pw_cbarg=0x55b505399918) at ../providers/implementations/storemgmt/file_store_any2obj.c:104
#17 0x00007fa59f0908cb in decoder_process (params=params@entry=0x0, arg=arg@entry=0x7fffd212c4d0) at ../crypto/encode_decode/decoder_lib.c:961
#18 0x00007fa59f090ba1 in OSSL_DECODER_from_bio (ctx=0x55b5053998e0, in=0x55b5053a7bf0) at ../crypto/encode_decode/decoder_lib.c:81
#19 0x00007fa59f1b08de in file_load_file (pw_cbarg=0x55b5053a7ce8, pw_cb=0x7fa59f0ef680 <ossl_pw_passphrase_callback_dec>, object_cbarg=0x7fffd212c630,
object_cb=0x7fa59f133270 <ossl_store_handle_load_result>, ctx=0x55b5053964d0) at ../providers/implementations/storemgmt/file_store.c:536
#20 file_load (loaderctx=0x55b5053964d0, object_cb=0x7fa59f133270 <ossl_store_handle_load_result>, object_cbarg=0x7fffd212c630,
pw_cb=0x7fa59f0ef680 <ossl_pw_passphrase_callback_dec>, pw_cbarg=0x55b5053a7ce8) at ../providers/implementations/storemgmt/file_store.c:704
#21 0x00007fa59f13184c in OSSL_STORE_load (ctx=0x55b5053a7ca0) at ../crypto/store/store_lib.c:428

The described behavior was observed on 3.0.x as on Ubuntu as well as on a self-compiled 3.1 static build.

This report was originally filed as security issue, however the project team considered this functional, hence the additional public report.

Reproduce the issue with the attached mac_sha_iter_maxint.p12 using both the x509 and verify commands. Start with providers/implementations/kdfs/pkcs12kdf.c and the PKCS#12 loading path in providers/implementations/storemgmt/store_result.c, using the supplied stack trace as a map. Determine the intended handling for extreme iteration counts and add coverage for the observed behavior.