Locale-encoded text is corrupted on FreeBSD, NetBSD, DragonFly BSD and macOS in non-UTF-8 locales

Bug report

On FreeBSD, NetBSD and DragonFly BSD, wchar_t values are not Unicode code points in non-UTF-8 locales: mbstowcs() and wcstombs() keep the raw locale values (e.g. the KOI8-U byte 0xD0 becomes 0x00D0, and the EUC-JP byte pair 0xB7 0xEE becomes 0xB7EE). CPython assumes that wchar_t is Unicode, so every interface which passes locale-encoded text through wchar_t produces mojibake.

For example, in the uk_UA.KOI8-U locale, time.strftime('%A') returns 'ÐÏÎÅĦÌÏË' instead of 'понеділок'. The same applies to locale.nl_langinfo(), locale.localeconv(), format(n, 'n'), localized OSError and ctypes error messages, sys.argv and readline history.

I propose to use iconv() in _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx() on these platforms. Unlike the codec machinery, iconv() is usable in early interpreter startup (e.g. for decoding the command line arguments), and the C library's iconv() supports all encodings used by its own locales. wcsftime() also cannot be used, so time.strftime() should use strftime() and decode its result. If iconv() is not functional for the locale encoding, the current behavior is kept.

Related: gh-93251 (localized gai_strerror() and hstrerror() messages are decoded as UTF-8 — affects all platforms, fixed separately).

Linked PRs

• gh-154722

Review _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx(), then inspect the time.strftime() path described in the report. Compare behavior in the listed BSD and macOS non-UTF-8 locales, including the fallback when iconv() is not functional. Done means the affected locale-facing interfaces no longer produce the reported mojibake.