PKCS#12 PBE doesn't convert to UTF-16BE correctly

See https://github.com/magnumripper/JohnTheRipper/issues/4179

The PKCS#12 PBE function cheats when "converting" to UTF-16BE. It simply casts char -> short, or "inserts every other zero". While this happens to be correct for ISO-8859-1, it's not for the remaining 99.8% of the Unicode charset or 99.5% of UCS-2.

Actually, as far as I can see, these functions can't ever take a "string" as input unless they also take a parameter describing the encoding of it (that, or requiring a certain encoding - which should probably be UTF-8 for guaranteeing full functionality) because otherwise you simply can't know how to convert it properly. Most other functions take binary data and length and then the caller can decide to convert [correctly] to UTF-16BE before calling them.

A super-quick "workaround" would be to clearly document that the function only work correctly for ASCII and ISO-8859-1. But then you wouldn't be compatible with libs that do it right.

This bug is particularly bad in that if you, say, encrypt a certificate on a system using mbedTLS and then try to decrypt it on a system without this bug, your password won't work. Or vice versa. We are seeing examples of this.

Start by locating the PKCS#12 PBE function that casts char to short, then read the linked JohnTheRipper issue and inspect how callers represent password strings. Define the supported input encoding and verify that non-ASCII passwords produce UTF-16BE-compatible encryption and decryption across implementations.

Written by the indexing model from the issue text.