indygreg / indygreg/PyOxidizer

Trying to use multiprocessing results in an infinite loop on windows 10

Open
#531 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
6.2k
Forks
256
PR merge metrics
No merged PRs in 30d

Description

I have run into an issue where creating a pool causes an infinite loop. After some testing, it seems not a function of the original script I was running, and was rather easy to recreate on the latest version.

TCL:

```
def make_exe():
dist = default_python_distribution()

policy = dist.make_python_packaging_policy()

python_config = dist.make_python_interpreter_config()
python_config.run_module = "testscript"

exe = dist.to_python_executable(
name="testbuild",
packaging_policy=policy,
config=python_config,
)

# Recursively scan the filesystem at 'path' and grab matching 'packages'
exe.add_python_resources(
exe.read_package_root(
path=".",
packages=["testscript"],
)
)

exe.windows_runtime_dlls_mode = "always"
#exe.tcl_files_path = "lib"
return exe

def make_embedded_resources(exe):
return exe.to_embedded_resources()

def make_install(exe):
# Create an object that represents our installed application file layout.
files = FileManifest()
# Add the generated executable to our install layout in the root directory.
files.add_python_resource(".", exe)
return files

register_target("exe", make_exe)
register_target(
"resources", make_embedded_resources, depends=["exe"], default_build_script=True
)
register_target("install", make_install, depends=["exe"], default=True)

resolve_targets()
```

Test script:
```
import platform
import sys
from multiprocessing import freeze_support, set_start_method, get_start_method,spawn,Pool

def f(x):
return x*x

def Main():
#I also tried a version where this was not in a seperate script, but rather in the 'if name' block
#set_start_method('spawn')
print(get_start_method())
try:
print(sys.frozen)
except:
print("False")
print(spawn.get_executable())
print(sys.executable)

print(platform.machine())
print(platform.version())
print(platform.platform())
print(platform.uname())
print(platform.system())
print(platform.processor())


with Pool() as pool:
print(pool.map(f, range(10)))

if __name__=="__main__":
freeze_support() #theoretically not needed I think? Doesn't help anyway.
Main()
```

Output:

```
spawn #correct start method

True #sys.frozen is True

...build\x86_64-pc-windows-msvc\debug\install\testbuild.exe #exe seems to be correct
...build\x86_64-pc-windows-msvc\debug\install\testbuild.exe
AMD64
10.0.19042
Windows-10-10.0.19042
uname_result(system='Windows', release='10', version='10.0.19042', machine='AMD64')
Windows
Intel64 Family 6 Model 126 Stepping 5, GenuineIntel

Traceback (most recent call last):
File "multiprocessing.spawn", line 107, in spawn_main
File "multiprocessing.reduction", line 79, in duplicate
TypeError: DuplicateHandle() argument 2 must be int, not dict
#with this traceback repeating forever until the process is killed
```

The actual error seems to be related to this:

https://bugs.python.org/issue38188

but it still seems to me that something odd is happening to produce so many errors, as well as something being incorrectly handled? If it is purely this error, anyone got any insight into how to monkey patch the pyoxidizer dist?

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Windows 10 build using the shown PyOxidizer script and testscript, then run the frozen executable with multiprocessing.Pool. Start with multiprocessing.spawn and multiprocessing.reduction at the reported DuplicateHandle traceback, and compare the behavior with the linked Python issue. Done means the packaged executable no longer repeats the traceback or hangs when the pool starts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.