Locale-encoded text is corrupted on FreeBSD, NetBSD, DragonFly BSD and macOS in non-UTF-8 locales

Bug report

On FreeBSD, NetBSD and DragonFly BSD, wchar_t values are not Unicode code points in non-UTF-8 locales: mbstowcs() and wcstombs() keep the raw locale values (e.g. the KOI8-U byte 0xD0 becomes 0x00D0, and the EUC-JP byte pair 0xB7 0xEE becomes 0xB7EE). CPython assumes that wchar_t is Unicode, so every interface which passes locale-encoded text through wchar_t produces mojibake.

For example, in the uk_UA.KOI8-U locale, time.strftime('%A') returns 'ÐÏÎÅĦÌÏË' instead of 'понеділок'. The same applies to locale.nl_langinfo(), locale.localeconv(), format(n, 'n'), localized OSError and ctypes error messages, sys.argv and readline history.

I propose to use iconv() in _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx() on these platforms. Unlike the codec machinery, iconv() is usable in early interpreter startup (e.g. for decoding the command line arguments), and the C library's iconv() supports all encodings used by its own locales. wcsftime() also cannot be used, so time.strftime() should use strftime() and decode its result. If iconv() is not functional for the locale encoding, the current behavior is kept.

Related: gh-93251 (localized gai_strerror() and hstrerror() messages are decoded as UTF-8 — affects all platforms, fixed separately).

Linked PRs

• gh-154722

Revisa _Py_DecodeLocaleEx() y _Py_EncodeLocaleEx(), y después inspecciona la ruta de time.strftime() descrita en el informe. Compara el comportamiento en las configuraciones regionales BSD y macOS sin UTF-8 indicadas, incluido el fallback cuando iconv() no funciona. Se considera terminado cuando las interfaces afectadas orientadas a la configuración regional ya no producen el mojibake indicado.