Expose XOFs like SHAKE128 and SHAKE256

SHAKE is a XOF from the Keccak family of algorithms. Loosely, you can consider them a digest. The thing that sets them apart though is that they have no fixed output size, instead callers indicate "How much data do you want from it?" The 128 and 256 are secure strengths, not output sizes like SHA-2-256.

Even though SHAKE is an EVP-MD, the ruby/openssl project assumes that all digests have a fixed length output.

digest = OpenSSL::Digest.new("SHAKE256") # Works
digest << "hello world" # Works
digest.digest # Fail: ArgumentError: negative string size (or size too big)

Largely because the gem is attempting to ask the EVP_MD how big the digest is using EVP_MD_CTX_size, which returns -1, which is where the ArgumentError comes from.

A small suggestion would be to recognize that when EVP_MD_CTX_size return -1, a length must be supplied to digest

digest = OpenSSL::Digest.new("SHAKE256")
digest << "hello world"
digest.digest(64)

This will allow using the SHAKE XOF.

As a further property of XOFs is "squeeze", exposed by OpenSSL's EVP_DigestSqueeze.

One could imagine doing something like

digest = OpenSSL::Digest.new("SHAKE256")
digest << "hello world"
digest.squeeze(32) + digest.squeeze(32) # Equivalent to digest(64) 

Squeeze allows someone to "read" from the XOF in pieces, instead of reading it all out at at once with digest.

Why is this useful?

SHAKE-256 is used to compute the message representative, or mu, when using ML-DSA in external mu. Ruby's ML-DSA does support signing mu instead of the message, but the lack of SHAKE makes it difficult to compute mu. For example, using a made-up mu:

key = OpenSSL::PKey.generate_key("ML-DSA-65")
mu = "\x00" * 64 # This should actually be computed with SHAKE-256
signature = key.sign(nil, mu, "mu" => "1")

Because SHAKE-256 can absorb very large amounts of data, it works like signing a hash with combined with external mu. This is how you can sign very large amounts of data, like a file, without putting the whole file in memory.

This is distinct, and different than HashML-DSA which has largely failed to gain traction in many other standards, and instead recommend using external-mu.

For the purposes of external mu, the digest.squeeze functionality is not needed with EVP_DigestSqueeze, but having some means digest(64) is needed.

I will offer my time to contribute the work, however I would like to better understand what the maintainers would want this to look like before getting started.

Start by tracing OpenSSL::Digest handling around EVP_MD_CTX_size and the digest method, then review how EVP_DigestSqueeze could fit the proposed API. Confirm the desired behavior for SHAKE128 and SHAKE256, including digest(length); done means variable-length output works without breaking fixed-length digests and is covered by tests.