[Feature Request] Add fork protection for GPU initialization for runtimes which aren't fork-safe
Nobody has claimed this yet.
- Dominant language
- Mojo
- Stars
- 29.8k
- Forks
- 3.2k
- PR merge metrics
- No merged PRs in 30d
Description
Summary
MAX should detect when Nvidia GPU operations are attempted in a forked child process and raise a clear error message, similar to PyTorch's fork protection mechanism. Currently, MAX allows GPU operations after fork(), which leads to segmentation faults and potential silent data corruption.
Problem
When a process forks after initializing MAX GPU devices, the child process can attempt GPU operations, leading to:
- Segmentation faults (SIGSEGV) - Child processes crash with exit status 139
- Silent data corruption - Some operations succeed before crashing, potentially corrupting GPU memory
- No error message - Users get a crash with no explanation of the cause
- Timing-dependent failures - Race conditions between parent and child GPU access
Minimal reproducible example
import os
from max.driver import Accelerator
from max.experimental.tensor import Tensor
# Initialize GPU in parent
gpu = Accelerator(0)
parent_tensor = Tensor.ones((10, 10), device=gpu)
print(f"[Parent] Tensor allocated on GPU")
# Fork the process
pid = os.fork()
if pid == 0:
# Child process
# This should raise an error but instead causes a segfault
child_tensor = Tensor.ones((5, 5), device=gpu)
print(f"[Child] Tensor allocated") # May or may not reach here
# Process exits with status 139 (SIGSEGV)
else:
# Parent process
import os
_, status = os.waitpid(pid, 0)
print(f"Child exited with status: {status}") # Shows 139
Current output:
[Parent] Tensor allocated on GPU
Child exited with status: 139
Desired output:
[Parent] Tensor allocated on GPU
RuntimeError: Cannot re-initialize MAX GPU in forked subprocess.
To use MAX with multiprocessing, you must use the 'spawn' start method.
How PyTorch solves this
PyTorch CUDA has excellent fork protection using POSIX pthread_atfork():
How it works
- Registration: When GPU is first initialized, PyTorch registers a
pthread_atfork()child handler - Automatic detection: When
fork()is called, the OS automatically calls the child handler in the forked process - Flag setting: The child handler sets
is_in_bad_fork = true - Early detection: Any subsequent GPU operation checks this flag and raises a clear error before attempting GPU work
- No crashes: The error is raised in Python before any invalid GPU operations occur
Different implementations
It seems the different backends/devices of pytorch have some kind of protection: https://github.com/search?q=repo%3Apytorch%2Fpytorch%20_is_in_bad_fork&type=code
Proposed solution for MAX
We can use Python's os.register_at_fork to avoid dropping to C-level pthread_atfork().
We have to find out if we have to do this only for nvidia GPUs or if we need to do this for other hardware accelerators as well.
Note that this won't protect pure Mojo programs from this forking issue if we do it in python.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Python MAX entry points shown in the reproducer: max.driver.Accelerator and max.experimental.tensor.Tensor, then review Python's os.register_at_fork behavior. Reproduce the fork-after-GPU-initialization case and determine whether protection applies only to Nvidia GPUs or other accelerators. Done means the forked child raises a clear RuntimeError before GPU work, while the issue's spawn guidance remains valid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100