IronLanguages / IronLanguages/ironpython3

Parallel execution of multiple IronPython engines with one runtime

Open
#1,826 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
2.8k
Forks
316
Avg merge
1d 9h
Merged PRs (30d)
1

Description

Description

Hi,
we found a changed behaviour in IP 3.4.1 compared to the IP 2.x implementation.
We are using IP to execute python code in multiple threads.
To improve performance the work threads are precreated and share the same (global) runtime.
After switching to IP 3 we found that addings an assembly via the clr methodes and also imports to e.g. the decimal module sometimes fail. This is different for what happend in IP 2.x, there was no issue, running using mutiple python enginens in parallel threads with one global runtime.

We have been able to mitigate the issue with the addition of a process wide lock object used in the methods responsible for imports and adding assembly references. However, we can't tell for sure that all potential methods are

Steps to Reproduce
  1. Create a shared runtime
  2. Create multipe thread having their private python engine
  3. Ensure that the threads are started at the same time
  4. Have the engine execute script code including clr.AddReference calls and "from decimal import Decimal"
  5. Attached is a C# sample program (renamed to .cs.txt to allow upload) and a patch file with the changes we made to try to fix the issue
    Program.cs.txt
    mutliThreadedSuport.patch

Expected behavior:
All threads execute the script code without errors

Actual behavior:
Some threads fail to execute the script code with error like "module 'numbers' not in sys.modules" or failure to add the assembly via the clr method. The actual error can be different.

Version Information
  • .NET platform used: .NET Framework
  • Version 4.6.2
  • Windows 11
  • 3.4.1 DEBUG (3.4.1.1000) / [.NETFramework,Version=v4.6.2 on .NET Framework 4.8.9282.0 (64-bit)]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached Program.cs.txt and reproduce the shared-runtime case using multiple private engines, concurrent clr.AddReference calls, and decimal imports. Review mutliThreadedSuport.patch alongside the reproduction to identify the affected import and assembly-reference paths; done means all threads execute the script without errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, python
Domain
backend, compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.