openssl / openssl/openssl

pkeyutl -sign -pkeyopt rsa_padding_mode:none rejects valid RSA modulus-sized input

Open
#30,777 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

issue: bug report
Dominant language
C
Stars
30.8k
Forks
11.5k
Avg merge
10m
Merged PRs (30d)
1

Description

I am trying to replace deprecated rsautl usage with pkeyutl. For RSA no-padding, the docs list rsa_padding_mode:none as a supported padding mode, but pkeyutl -sign does not accept valid modulus-sized input and seems to enforce digest-style constraints instead.
For RSA-2048, a 256-byte input block should be valid for raw RSA/no-padding, but this fails:

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out key.pem
printf 'A%.0s' {1..256} > in-256.bin
openssl pkeyutl -sign -pkeyopt rsa_padding_mode:none -inkey key.pem -in in-256.bin -out sig.bin

I have tried different input block sizes out of curiosity, and here's what I found:

  • 20, 32, 64 bytes: RSA_padding_add_none:data too small for key size
  • 128, 255, 256 bytes: Error: The input data looks too long to be a hash

In https://github.com/openssl/openssl/blob/master/apps/pkeyutl.c#L532, non-raw sign/verify input is rejected if buf_inlen > EVP_MAX_MD_SIZE. With EVP_MAX_MD_SIZE == 64, that explains the 128+ byte failure, but it also means pkeyutl -sign cannot be used for valid raw RSA/no-padding blocks.

I have found an ugly workaround that actually works but I would rather keep using rsautl for the sake of code clarity:

openssl pkeyutl -decrypt -pkeyopt rsa_padding_mode:none -inkey key.pem -in in-256.bin -out out.bin

Questions:

  • Is rsa_padding_mode:none supposed to be supported for pkeyutl -sign?
  • If yes, should modulus-sized input be accepted without the digest-length check?
  • If no, could the documentation clarify that pkeyutl -sign is not a replacement for raw RSA/private-key no-padding operations previously done via rsautl -sign -raw?

Version: OpenSSL 3.0.13 30 Jan 2024 on Ubuntu 24.04

See also: https://github.com/openssl/openssl/issues/9658

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the RSA-2048 commands in the issue, then inspect apps/pkeyutl.c around line 532 and compare the behavior with issue 9658. Determine whether rsa_padding_mode:none with pkeyutl -sign should accept modulus-sized input; completion should establish the intended behavior and update the implementation or documentation accordingly.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
cli, cryptography, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.