OpenSSL::PKey::RSA.generate always returns a OpenSSL::PKey::RSA, even when sublcassed, on Ruby 3.1/openssl-3.0.0

I noticed a change in behavior for OpenSSL::PKey::RSA.generate between Ruby 3.0 (openssl-2.x) and ruby 3.1 (openssl-3.0.0). In Ruby 3.1 OpenSSL::PKey::RSA.generate will always return a OpenSSL::PKey::RSA instance, even when the class has been sublcassed.

Steps To Reproduce

require 'openssl'

class MyRSA < OpenSSL::PKey::RSA
end

p MyRSA.generate(1024)

Expected Behavior

#<MyRSA:...>

Actual Behavior

#<OpenSSL::PKey::RSA:...>

Version

• Ruby 3.0 (Works):
  • ruby 3.0.4p208 (2022-04-12 revision 3fa771dded) [x86_64-linux]
  • `openssl (default: 2.2.1)
• Ruby 3.1 (Fails):
  • ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
  • openssl (default: 3.0.0)

This regression seems to have appeared in 3.0.0.

Start by reproducing the issue with the OpenSSL::PKey::RSA.generate entry point and the MyRSA subclass shown in the report. Trace how generate constructs its return value in the Ruby OpenSSL extension, then add coverage confirming the subclass is preserved and run the relevant OpenSSL tests.