modular / modular/modular

[Feature Request] Add fork protection for GPU initialization for runtimes which aren't fork-safe

Open
#5,483 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement gex-graph-api graph-api max
Dominant language
Mojo
Stars
29.8k
Forks
3.2k
PR merge metrics
No merged PRs in 30d

Description

Summary

MAX should detect when Nvidia GPU operations are attempted in a forked child process and raise a clear error message, similar to PyTorch's fork protection mechanism. Currently, MAX allows GPU operations after fork(), which leads to segmentation faults and potential silent data corruption.

Problem

When a process forks after initializing MAX GPU devices, the child process can attempt GPU operations, leading to:

  1. Segmentation faults (SIGSEGV) - Child processes crash with exit status 139
  2. Silent data corruption - Some operations succeed before crashing, potentially corrupting GPU memory
  3. No error message - Users get a crash with no explanation of the cause
  4. Timing-dependent failures - Race conditions between parent and child GPU access
Minimal reproducible example
import os
from max.driver import Accelerator
from max.experimental.tensor import Tensor

# Initialize GPU in parent
gpu = Accelerator(0)
parent_tensor = Tensor.ones((10, 10), device=gpu)
print(f"[Parent] Tensor allocated on GPU")

# Fork the process
pid = os.fork()

if pid == 0:
    # Child process
    # This should raise an error but instead causes a segfault
    child_tensor = Tensor.ones((5, 5), device=gpu)
    print(f"[Child] Tensor allocated")  # May or may not reach here
    # Process exits with status 139 (SIGSEGV)
else:
    # Parent process
    import os
    _, status = os.waitpid(pid, 0)
    print(f"Child exited with status: {status}")  # Shows 139

Current output:

[Parent] Tensor allocated on GPU
Child exited with status: 139

Desired output:

[Parent] Tensor allocated on GPU
RuntimeError: Cannot re-initialize MAX GPU in forked subprocess.
To use MAX with multiprocessing, you must use the 'spawn' start method.

How PyTorch solves this

PyTorch CUDA has excellent fork protection using POSIX pthread_atfork():

How it works
  1. Registration: When GPU is first initialized, PyTorch registers a pthread_atfork() child handler
  2. Automatic detection: When fork() is called, the OS automatically calls the child handler in the forked process
  3. Flag setting: The child handler sets is_in_bad_fork = true
  4. Early detection: Any subsequent GPU operation checks this flag and raises a clear error before attempting GPU work
  5. No crashes: The error is raised in Python before any invalid GPU operations occur
Different implementations

It seems the different backends/devices of pytorch have some kind of protection: https://github.com/search?q=repo%3Apytorch%2Fpytorch%20_is_in_bad_fork&type=code

Proposed solution for MAX

We can use Python's os.register_at_fork to avoid dropping to C-level pthread_atfork().
We have to find out if we have to do this only for nvidia GPUs or if we need to do this for other hardware accelerators as well.

Note that this won't protect pure Mojo programs from this forking issue if we do it in python.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Python MAX entry points shown in the reproducer: max.driver.Accelerator and max.experimental.tensor.Tensor, then review Python's os.register_at_fork behavior. Reproduce the fork-after-GPU-initialization case and determine whether protection applies only to Nvidia GPUs or other accelerators. Done means the forked child raises a clear RuntimeError before GPU work, while the issue's spawn guidance remains valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.