modular / modular/modular

[BUG][stdlib] Limitation of `parallelize` with Mojo code that interacts with Python

Open
#3,993 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug mojo mojo-repo
Dominant language
Mojo
Stars
29.8k
Forks
3.2k
PR merge metrics
No merged PRs in 30d

Description

Bug description
Actual behavior

When a parallelized function call interacts with Python, it crashes at runtime under certain conditions. Referring to the code example below, struct Bar.start() uses parallelize to execute Foo.start() on instances stored in a variable of type List[Foo]. The wheels fall off if num_work_items and num_workers each is greater than 1, and the function interacts with Python. Otherwise, num_workers greater than 1 works as expected with Mojo-only code in Foo.start().

Expected behavior

Parallelize works with num_workers greater than 1 in Mojo code that interacts with Python.

Discord

Original post
https://discord.com/channels/1087530497313357884/1151418092052815884/1329614332744564787
Feedback from Owen Hilyard and Joe Loser
https://discord.com/channels/1087530497313357884/1151418092052815884/1330181446345953302
https://discord.com/channels/1087530497313357884/1151418092052815884/1330190214035017768

Steps to reproduce
from algorithm.functional import parallelize
from python import Python, PythonObject


@value
struct Foo(Copyable & Movable):
    var name: String
    var py_uuid: PythonObject

    fn __init__(out self, name: String) raises:
        self.name = name
        self.py_uuid = Python.import_module("uuid")

    fn start(self, out result: PythonObject):
        try:
            _uuid = self.py_uuid.uuid4()
        except e:
            print("start:", e)
            _uuid = PythonObject("<no uuid>")

        result = _uuid


alias Foos = List[Foo]


@value
struct Bar(Copyable & Movable):
    var name: String
    var _foos: Foos

    fn __init__(out self, name: String):
        self.name = name
        self._foos = Foos()

    fn append(mut self, foo: Foo):
        self._foos.append(foo)

    fn start(self):
        # Serial: This works - OK
        # for index in range(len(self._foos)):
        #     print(self._foos[index].start())

        # Parallel: If num_workers > 1 - KO
        @parameter
        fn parallelize_start(index: Int):
            print("parallelize_start:", index)
            print(self._foos[index].start())

        num_work_items = len(self._foos)
        # num_workers = 1  # OK - Essentially Serial
        num_workers = 2  # KO
        # num_workers = num_work_items  # KO
        parallelize[parallelize_start](num_work_items, num_workers)
        print("Parallel start - Done!")


fn main() raises:
    bars = Bar("My Foos")
    bars.append(Foo("One"))
    bars.append(Foo("Two"))
    bars.start()
System information

Package Version Build Size Kind Source
max 25.1.0.dev2025021205 release 9.7 KiB conda max
max-core 25.1.0.dev2025021205 release 231 MiB conda max-core
max-python 25.1.0.dev2025021205 release 113.9 MiB conda max-python

 Magic version: 0.6.4

System

   Pixi version: 0.40.3
       Platform: linux-64

Virtual packages: __unix=0=0
: __linux=6.11.0=0
: __glibc=2.40=0
: __cuda=12.6=0
: __archspec=1=skylake
Cache dir: /home/johnsoe1/.cache/rattler/cache
Auth storage: /home/johnsoe1/.rattler/credentials.json
Config locations: No config files found

Global

        Bin dir: /home/johnsoe1/.modular/bin
Environment dir: /home/johnsoe1/.modular/envs
   Manifest dir: /home/johnsoe1/.modular/manifests/pixi-global.toml

Project

           Name: EzAsanaDL
        Version: 0.1.0
  Manifest file: /home/johnsoe1/dev/EzProjects/EzAsanaDL/mojoproject.toml
   Last updated: 12-02-2025 09:19:49

Environments

    Environment: default
       Features: default
       Channels: conda-forge, https://conda.modular.com/max-nightly

Dependency count: 7
Dependencies: max, python, pycryptodome, screeninfo, pytest, pip, jupyter
PyPI Dependencies: asana, timeloop-ng, dearpygui
Target platforms: linux-64
Tasks: tests

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the provided Foo and Bar example, using algorithm.functional.parallelize and Python.import_module("uuid"). Then inspect the parallelize implementation and its interaction with PythonObject values. Done means the example completes with multiple work items and workers without a runtime crash, while Mojo-only behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.