Inconsistent handling of non-ASCII characters in encodings.normalize_encoding()
Open
Nobody has claimed this yet.
3.13
3.14
3.15
stdlib
topic-unicode
type-bug
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 36k
- PR merge metrics
- PR metrics pending
Description
Bug report
#83518 changed handling of non-ASCII characters in encodings.normalize_encoding(), but it is still inconsistent with codecs.lookup(), and not even self-consistent. For example:
>>> import encodings
>>> encodings.normalize_encoding('a¤b')
'a_b'
>>> encodings.normalize_encoding('aæb')
'ab'
>>> encodings.normalize_encoding('a-¤')
'a'
>>> encodings.normalize_encoding('a-æ')
'a_'
>>> encodings.normalize_encoding('a-¤-b')
'a_b'
>>> encodings.normalize_encoding('a-æ-b')
'a__b'
You can even get an underscore at the end or repeated underscores in the middle.
cc @malemburg, @vstinner, @shihai1991
Linked PRs
- gh-136737
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with encodings.normalize_encoding() and compare its non-ASCII handling with codecs.lookup(), using the examples in this issue to reproduce trailing and repeated underscores. Done means the normalization behavior is consistent with codecs.lookup() and internally consistent; linked PR gh-136737 indicates work is already underway.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- internationalization
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100