python / python/cpython

Windows code page codec cannot encode with code pages that require dwFlags=0

Open
#155,016 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

3.13 3.14 3.15 3.16 OS-windows topic-unicode type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

On Windows, codecs.code_page_encode() fails with OSError: [WinError 87] The parameter is incorrect for every code page that WideCharToMultiByte() requires to be called with dwFlags = 0 and lpUsedDefaultChar = NULL: 50220, 50221, 50222, 50225, 50227, 50229, 52936 (HZ-GB2312), 54936 (GB18030), 57002-57011 (ISCII), 65000 and 42. Decoding these code pages works.

>>> import codecs
>>> codecs.code_page_decode(50220, b'\x1b$B$"\x1b(B')
('あ', 8)
>>> codecs.code_page_encode(50220, 'あ')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [WinError 87] The parameter is incorrect

encode_code_page_flags() returns WC_NO_BEST_FIT_CHARS, and encode_code_page_strict() passes a non-NULL lpUsedDefaultChar, for every code page except CP_UTF8 and CP_UTF7. decode_code_page_strict() retries with flags = 0 when MultiByteToWideChar() fails with ERROR_INVALID_FLAGS; the encoder has no such fallback. With dwFlags = 0 and lpUsedDefaultChar = NULL the conversion works:

>>> import ctypes
>>> buf = ctypes.create_string_buffer(64)
>>> n = ctypes.windll.kernel32.WideCharToMultiByte(50220, 0, 'あ', 1, buf, 64, None, None)
>>> buf.raw[:n]
b'\x1b$B$"\x1b(B'

Note that usedDefaultChar is how the encoder detects unencodable characters, so dropping it needs a replacement (for example decoding the result back and comparing).

cp51932, cp51936 and cp51950 fail in both directions; those code pages are not supported by MultiByteToWideChar()/WideCharToMultiByte() at all.

Linked PRs
  • gh-155018

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with codecs.code_page_encode() and the encode_code_page_flags() and encode_code_page_strict() helpers, then compare their behavior with decode_code_page_strict(). Reproduce the failure for code page 50220 and verify the listed Windows code pages encode successfully while unencodable-character detection remains covered. Linked PR gh-155018 indicates that work is already underway.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.