pkeyutl -sign -pkeyopt rsa_padding_mode:none rejects valid RSA modulus-sized input
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 30.8k
- Forks
- 11.5k
- Avg merge
- 10m
- Merged PRs (30d)
- 1
Description
I am trying to replace deprecated rsautl usage with pkeyutl. For RSA no-padding, the docs list rsa_padding_mode:none as a supported padding mode, but pkeyutl -sign does not accept valid modulus-sized input and seems to enforce digest-style constraints instead.
For RSA-2048, a 256-byte input block should be valid for raw RSA/no-padding, but this fails:
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out key.pem
printf 'A%.0s' {1..256} > in-256.bin
openssl pkeyutl -sign -pkeyopt rsa_padding_mode:none -inkey key.pem -in in-256.bin -out sig.bin
I have tried different input block sizes out of curiosity, and here's what I found:
- 20, 32, 64 bytes: RSA_padding_add_none:data too small for key size
- 128, 255, 256 bytes: Error: The input data looks too long to be a hash
In https://github.com/openssl/openssl/blob/master/apps/pkeyutl.c#L532, non-raw sign/verify input is rejected if buf_inlen > EVP_MAX_MD_SIZE. With EVP_MAX_MD_SIZE == 64, that explains the 128+ byte failure, but it also means pkeyutl -sign cannot be used for valid raw RSA/no-padding blocks.
I have found an ugly workaround that actually works but I would rather keep using rsautl for the sake of code clarity:
openssl pkeyutl -decrypt -pkeyopt rsa_padding_mode:none -inkey key.pem -in in-256.bin -out out.bin
Questions:
- Is rsa_padding_mode:none supposed to be supported for pkeyutl -sign?
- If yes, should modulus-sized input be accepted without the digest-length check?
- If no, could the documentation clarify that pkeyutl -sign is not a replacement for raw RSA/private-key no-padding operations previously done via rsautl -sign -raw?
Version: OpenSSL 3.0.13 30 Jan 2024 on Ubuntu 24.04
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the RSA-2048 commands in the issue, then inspect apps/pkeyutl.c around line 532 and compare the behavior with issue 9658. Determine whether rsa_padding_mode:none with pkeyutl -sign should accept modulus-sized input; completion should establish the intended behavior and update the implementation or documentation accordingly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- cli, cryptography, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100