[BUG][stdlib] Limitation of `parallelize` with Mojo code that interacts with Python
Nobody has claimed this yet.
- Dominant language
- Mojo
- Stars
- 29.8k
- Forks
- 3.2k
- PR merge metrics
- No merged PRs in 30d
Description
Bug description
Actual behavior
When a parallelized function call interacts with Python, it crashes at runtime under certain conditions. Referring to the code example below, struct Bar.start() uses parallelize to execute Foo.start() on instances stored in a variable of type List[Foo]. The wheels fall off if num_work_items and num_workers each is greater than 1, and the function interacts with Python. Otherwise, num_workers greater than 1 works as expected with Mojo-only code in Foo.start().
Expected behavior
Parallelize works with num_workers greater than 1 in Mojo code that interacts with Python.
Discord
Original post
https://discord.com/channels/1087530497313357884/1151418092052815884/1329614332744564787
Feedback from Owen Hilyard and Joe Loser
https://discord.com/channels/1087530497313357884/1151418092052815884/1330181446345953302
https://discord.com/channels/1087530497313357884/1151418092052815884/1330190214035017768
Steps to reproduce
from algorithm.functional import parallelize
from python import Python, PythonObject
@value
struct Foo(Copyable & Movable):
var name: String
var py_uuid: PythonObject
fn __init__(out self, name: String) raises:
self.name = name
self.py_uuid = Python.import_module("uuid")
fn start(self, out result: PythonObject):
try:
_uuid = self.py_uuid.uuid4()
except e:
print("start:", e)
_uuid = PythonObject("<no uuid>")
result = _uuid
alias Foos = List[Foo]
@value
struct Bar(Copyable & Movable):
var name: String
var _foos: Foos
fn __init__(out self, name: String):
self.name = name
self._foos = Foos()
fn append(mut self, foo: Foo):
self._foos.append(foo)
fn start(self):
# Serial: This works - OK
# for index in range(len(self._foos)):
# print(self._foos[index].start())
# Parallel: If num_workers > 1 - KO
@parameter
fn parallelize_start(index: Int):
print("parallelize_start:", index)
print(self._foos[index].start())
num_work_items = len(self._foos)
# num_workers = 1 # OK - Essentially Serial
num_workers = 2 # KO
# num_workers = num_work_items # KO
parallelize[parallelize_start](num_work_items, num_workers)
print("Parallel start - Done!")
fn main() raises:
bars = Bar("My Foos")
bars.append(Foo("One"))
bars.append(Foo("Two"))
bars.start()
System information
Package Version Build Size Kind Source
max 25.1.0.dev2025021205 release 9.7 KiB conda max
max-core 25.1.0.dev2025021205 release 231 MiB conda max-core
max-python 25.1.0.dev2025021205 release 113.9 MiB conda max-python
Magic version: 0.6.4
System
Pixi version: 0.40.3
Platform: linux-64
Virtual packages: __unix=0=0
: __linux=6.11.0=0
: __glibc=2.40=0
: __cuda=12.6=0
: __archspec=1=skylake
Cache dir: /home/johnsoe1/.cache/rattler/cache
Auth storage: /home/johnsoe1/.rattler/credentials.json
Config locations: No config files found
Global
Bin dir: /home/johnsoe1/.modular/bin
Environment dir: /home/johnsoe1/.modular/envs
Manifest dir: /home/johnsoe1/.modular/manifests/pixi-global.toml
Project
Name: EzAsanaDL
Version: 0.1.0
Manifest file: /home/johnsoe1/dev/EzProjects/EzAsanaDL/mojoproject.toml
Last updated: 12-02-2025 09:19:49
Environments
Environment: default
Features: default
Channels: conda-forge, https://conda.modular.com/max-nightly
Dependency count: 7
Dependencies: max, python, pycryptodome, screeninfo, pytest, pip, jupyter
PyPI Dependencies: asana, timeloop-ng, dearpygui
Target platforms: linux-64
Tasks: tests
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the provided Foo and Bar example, using algorithm.functional.parallelize and Python.import_module("uuid"). Then inspect the parallelize implementation and its interaction with PythonObject values. Done means the example completes with multiple work items and workers without a runtime crash, while Mojo-only behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100