IronLanguages / IronLanguages/ironpython3

CJK codecs are incomplete.

Open
#716 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
2.8k
Forks
316
Avg merge
1d 9h
Merged PRs (30d)
1

Description

Description

For CJK codecs (cp932 for example), the "encoding" sysmodule requires some complements and they are slightly complex.
In CPython, these complements are implemented in C.

Steps to Reproduce
  1. Run ipy.exe without arguments.

Expected behavior:
IronPython.Runtime.Operations.PythonOps.LookupEncoding() does not throw any exceptions.
This method is called by importing "site" module.

Actual behavior:
IronPython.Runtime.Operations.PythonOps.LookupEncoding() throws following exception:
PythonOps.LookupError("unknown encoding: {0}", encoding)

Versions

IronPython 3.4.0a0 DEBUG (3.4.0.0000)
[.NETFramework,Version=v4.5 on .NET Framework 4.8.4075.0 (64-bit)] on win32

$ git rev-parse HEAD
342e49f69d9a967606d49394df51a059a55a4381

Workarounds (for developers)

The immediate issue is that make.ps1 test-all fails on CJK environment, but this can be workaround by (locally) applying patch as follows:

-                return "cp" + CultureInfo.CurrentCulture.TextInfo.ANSICodePage.ToString();
+                return "cp" + (new CultureInfo("en-US", false)).TextInfo.ANSICodePage.ToString();

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at IronPython.Runtime.Operations.PythonOps.LookupEncoding() and reproduce the failure by running ipy.exe without arguments, then run make.ps1 test-all in a CJK environment. Compare the relevant CPython Modules/cjkcodecs files, including _codecs_jp.c, to identify the missing complements. Done means importing site and the test suite no longer fail with an unknown encoding error.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, csharp, python
Domain
backend, internationalization
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.