Locale-encoded text is corrupted on FreeBSD, NetBSD, DragonFly BSD and macOS in non-UTF-8 locales

Bug report

On FreeBSD, NetBSD and DragonFly BSD, wchar_t values are not Unicode code points in non-UTF-8 locales: mbstowcs() and wcstombs() keep the raw locale values (e.g. the KOI8-U byte 0xD0 becomes 0x00D0, and the EUC-JP byte pair 0xB7 0xEE becomes 0xB7EE). CPython assumes that wchar_t is Unicode, so every interface which passes locale-encoded text through wchar_t produces mojibake.

For example, in the uk_UA.KOI8-U locale, time.strftime('%A') returns 'ÐÏÎÅĦÌÏË' instead of 'понеділок'. The same applies to locale.nl_langinfo(), locale.localeconv(), format(n, 'n'), localized OSError and ctypes error messages, sys.argv and readline history.

I propose to use iconv() in _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx() on these platforms. Unlike the codec machinery, iconv() is usable in early interpreter startup (e.g. for decoding the command line arguments), and the C library's iconv() supports all encodings used by its own locales. wcsftime() also cannot be used, so time.strftime() should use strftime() and decode its result. If iconv() is not functional for the locale encoding, the current behavior is kept.

Related: gh-93251 (localized gai_strerror() and hstrerror() messages are decoded as UTF-8 — affects all platforms, fixed separately).

Linked PRs

• gh-154722

Esamina _Py_DecodeLocaleEx() e _Py_EncodeLocaleEx(), quindi ispeziona il percorso di time.strftime() descritto nel report. Confronta il comportamento nelle impostazioni locali BSD e macOS non-UTF-8 elencate, incluso il fallback quando iconv() non è operativo. Il lavoro è completato quando le interfacce interessate rivolte alle impostazioni locali non producono più il mojibake segnalato.