google-deepmind / google-deepmind/mujoco
Studio native viewer cannot start on macOS: NSWindow created off the main thread
- Dominant language
- C++
- Stars
- 15.2k
- Forks
- 1.8k
- Avg merge
- 10d 16h
- Merged PRs (30d)
- 25
Description
### Intro
Hi!
I use MuJoCo for headless physics simulation and offscreen rendering distributed across many workers, and I have been scripting the viewer to drive job submission. That is what led me into experimental/studio.
### My setup
- MuJoCo 3.13.0, installed from the PyPI wheel
- API: Python
- Architecture: arm64 (Apple M2 Pro)
- OS: macOS 26.6.2
- Python 3.11.5
### What's happening? What did you expect?
`launch_passive` and `launch_native` create the viewer window on a worker thread.
macOS requires `NSWindow` to be created on the process main thread, so on macOS
the window is never created and the viewer does not run.
What makes this hard to handle rather than merely broken:
1. **No exception is raised.** An error is printed to stderr by native code, and
`launch_passive` returns normally.
2. **`handle.is_running()` returns `True`** immediately after the failure, so a
caller polling it is actively misled.
3. The process then terminates with exit status 1 during the first `handle.sync()`
loop. `print("finished normally")` at the end of the script never runs.
Observed output:
```
launch_passive returned, is_running: True
ERROR: Error creating window: NSWindow should only be instantiated on the main thread!
```
Exit status 1.
I expected either a working viewer window, or a raised exception identifying the
platform limitation so that a caller can fall back.
### Cause
`launch_native` delegates to `launch_thread`, which puts the viewer on a worker
thread and leaves the sim on the main thread:
```python
# python/mujoco/experimental/studio/launch_thread.py
def launch_thread(target_fn, *, sim_plugins=None):
viewer_endpoint, sim_endpoint = make_thread_endpoints()
thread = threading.Thread(target=target_fn, args=(viewer_endpoint,), daemon=True)
thread.start()
```
`target_fn` is the viewer. That arrangement is fine on X11 and Windows, and cannot
work on macOS, where AppKit only services UI objects created on the main thread.
The classic viewer solves the same problem with the `mjpython` launcher, which
keeps the Cocoa run loop on the main thread; Studio has no equivalent.
### Suggested fix
Two options, both inside `launch_native`:
1. Provide a blocking main-thread launcher alongside the threaded one, for callers
that do not need to keep control of the main thread. This works on every
platform, so it need not be macOS-specific.
2. Detect macOS in `launch_passive`/`launch_native` and invert the threading. This
changes the contract, since the call becomes blocking, so raising a clear error
that points at a blocking launcher may be preferable to changing behaviour
silently.
In either case, having the current path raise rather than print and exit would let
a caller detect the condition.
A working implementation of option 1 is in the last code block below. I am happy to
open a PR for whichever shape you prefer.
### Steps for reproduction
1. `pip install mujoco==3.13.0` on macOS.
2. Save `minimal.xml` and `repro.py` below into the same directory.
3. `python repro.py`
No `mjpython` is involved; the same result occurs under `mjpython`.
### Minimal model for reproduction
`minimal.xml`, loadable as-is, no binary assets required:
```xml
```
### Code required for reproduction
`repro.py`:
```python
import time
import mujoco
from mujoco.experimental.studio import launch_passive, messages, viewer_protocol
model = mujoco.MjModel.from_xml_path("minimal.xml")
data = mujoco.MjData(model)
handle = launch_passive.launch_passive(
viewer_protocol.ViewerConfig(title="studio repro", width=900, height=600)
)
handle.send_to_viewer(messages.ModelEvent(model=model, path="minimal.xml"))
print("launch_passive returned, is_running:", handle.is_running(), flush=True)
for _ in range(200):
model, data = handle.sync(model, data)
time.sleep(0.02)
print("finished normally", flush=True)
```
### Working around it, and a candidate fix
Studio already separates the two sides over channels and exposes everything needed
to wire them the other way round, so no change to MuJoCo is required to get a
working viewer today. Running the viewer on the main thread and the sim on a
worker thread works on macOS with plain `python`:
```python
import threading
from mujoco.experimental.studio import (launch_native, launch_thread, messages,
step_control, viewer_app, viewer_handle,
viewer_protocol)
def launch_on_main_thread(model, data, *, scene_path="", viewer_plugins=(),
sim_plugins=None, config=None):
config = config or viewer_protocol.ViewerConfig()
sim_plugins = sim_plugins or [step_control.StepControl()]
plugins = [viewer_app.ViewerApp(), *viewer_plugins]
viewer_endpoint, sim_endpoint = launch_thread.make_thread_endpoints()
closed = threading.Event()
handle = viewer_handle.ViewerHandle(
sim_endpoint, sim_plugins=list(sim_plugins),
is_alive_fn=lambda: not closed.is_set(), shutdown_fn=lambda: None)
def sim_loop():
m, d = model, data
handle.send_to_viewer(messages.ModelEvent(model=m, path=scene_path))
while not closed.is_set():
m, d = handle.sync(m, d)
thread = threading.Thread(target=sim_loop, daemon=True)
thread.start()
try:
# Blocks. This is the call that has to be on the main thread.
launch_native.run_native_viewer(config, viewer_endpoint, plugins=plugins)
finally:
closed.set()
thread.join(timeout=5)
```
Verified on the setup above: the window opens, Filament resolves to
`[Apple], [Apple M2 Pro], [4.1 Metal - 90.5]`, plugins receive `ModelEvent` and
`BuildGuiEvent`, and it sustains roughly 140 fps.
### Confirmations
- [x] I searched the [latest documentation](https://mujoco.readthedocs.io/en/latest/overview.html) thoroughly before posting.
- [x] I searched previous [Issues](https://github.com/google-deepmind/mujoco/issues) and [Discussions](https://github.com/google-deepmind/mujoco/discussions), I am certain this has not been raised before.
Contributor guide
Research direction
Reproduce the failure with repro.py, then read python/mujoco/experimental/studio/launch_thread.py and the launch_native and launch_passive entry points. Compare that path with run_native_viewer and the reported main-thread workaround. Done means the macOS viewer opens reliably or the caller receives a clear detectable error instead of a misleading running state and exit status 1.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, python
- Domain
- desktop
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100