IronLanguages / IronLanguages/ironpython3
Properly handle preferred encoding
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 2.8k
- Forks
- 316
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 1
Description
The builtin open method uses the preferred encoding (which can be obtained via locale.getpreferredencoding(False)) when opening a text file without specifying an encoding. On Linux/macOS this was incorrectly returning Windows code pages. https://github.com/IronLanguages/ironpython3/pull/1528 changes this always return UTF-8 on non-Windows systems. This should be updated to use the proper logic instead.
Additionally, on Windows, Encoding.GetEncoding(0).CodePage is used to get the the ACP which while probably correct, may be overridden via an EncodingProvider. It may be better to rely on GetACP directly...
See https://github.com/IronLanguages/ironpython3/issues/1527 for additional discussion.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the builtin open implementation and review PR 1528 alongside issue 1527, then compare its encoding selection with locale.getpreferredencoding(False). Check the Windows path involving Encoding.GetEncoding(0).CodePage and GetACP; done means text files without an explicit encoding use the proper platform-specific preferred encoding.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, python
- Domain
- backend, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100