Locale-encoded text is corrupted on FreeBSD, NetBSD, DragonFly BSD and macOS in non-UTF-8 locales

Bug report

On FreeBSD, NetBSD and DragonFly BSD, wchar_t values are not Unicode code points in non-UTF-8 locales: mbstowcs() and wcstombs() keep the raw locale values (e.g. the KOI8-U byte 0xD0 becomes 0x00D0, and the EUC-JP byte pair 0xB7 0xEE becomes 0xB7EE). CPython assumes that wchar_t is Unicode, so every interface which passes locale-encoded text through wchar_t produces mojibake.

For example, in the uk_UA.KOI8-U locale, time.strftime('%A') returns 'ÐÏÎÅĦÌÏË' instead of 'понеділок'. The same applies to locale.nl_langinfo(), locale.localeconv(), format(n, 'n'), localized OSError and ctypes error messages, sys.argv and readline history.

I propose to use iconv() in _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx() on these platforms. Unlike the codec machinery, iconv() is usable in early interpreter startup (e.g. for decoding the command line arguments), and the C library's iconv() supports all encodings used by its own locales. wcsftime() also cannot be used, so time.strftime() should use strftime() and decode its result. If iconv() is not functional for the locale encoding, the current behavior is kept.

Related: gh-93251 (localized gai_strerror() and hstrerror() messages are decoded as UTF-8 — affects all platforms, fixed separately).

Linked PRs

• gh-154722

_Py_DecodeLocaleEx() と _Py_EncodeLocaleEx() を確認し、その後、レポートに記載された time.strftime() の経路を調査してください。記載されている BSD および macOS の非 UTF-8 ロケールでの動作を、iconv() が機能しない場合のフォールバックも含めて比較してください。影響を受けるロケール関連インターフェースで、報告された文字化けが発生しなくなれば完了です。