NVIDIA-NeMo / NVIDIA-NeMo/RL

Worker startup fails when the repo path contains a space (unquoted git_root re-parsed by shlex.split)

Open
#3,800 0 comments 0 reactions 1 assignee Claimed by @kajalj22 View on GitHub
Automation bug community-request
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

## What happens

Any run dies during worker startup when the repo lives under a path containing a space. The failure surfaces as a Ray actor-creation error, which does not point at the path:

```
subprocess.CalledProcessError: Command '['uv', 'run', '--locked', '--extra', 'vllm',
'--directory', '/mnt/c/Users/Tianyi', 'Zhang/.../RL', 'echo', 'Finished creating venv ...']'
returned non-zero exit status 2.
```

Note `'--directory', '/mnt/c/Users/Tianyi', 'Zhang/.../RL'` — the path became two arguments.

## Why

`virtual_cluster.py` builds the executable as a command *string* with the path interpolated unquoted:

```python
# nemo_rl/distributed/virtual_cluster.py:59-80
BASE = f"uv run --locked --directory {git_root}"
VLLM = f"uv run --locked --extra vllm --directory {git_root}"
FSDP = f"uv run --locked --extra fsdp --directory {git_root}"
AUTOMODEL = f"uv run --locked --extra automodel --directory {git_root}"
MCORE = f"uv run --locked --extra mcore --directory {git_root}"
NEMO_GYM = f"uv run --locked --extra nemo_gym --directory {git_root}"
SGLANG = f"uv run --locked --extra sglang --directory {git_root}"
TRTLLM = f"uv run --locked --extra trtllm --directory {git_root}"
```

and `nemo_rl/modelopt/registry.py:24,29,34` does the same three more times — 11 sites in all.

That string is then re-parsed as a shell word list:

```python
# nemo_rl/utils/venvs.py:92
exec_cmd = shlex.split(py_executable)
```

so any space in `git_root` becomes an argument boundary.

## Repro

No NeMo-RL import needed; this is the two lines verbatim.

```python
import shlex
for git_root in ("/home/u/RL", "/mnt/c/Users/Tianyi Zhang/scratchpad/RL", "/Users/First Last/src/RL"):
s = f"uv run --locked --extra vllm --directory {git_root}"
parts = shlex.split(s)
print(git_root, "->", parts[parts.index("--directory") + 1])
```

```
/home/u/RL -> /home/u/RL
/mnt/c/Users/Tianyi Zhang/scratchpad/RL -> /mnt/c/Users/Tianyi <-- truncated
/Users/First Last/src/RL -> /Users/First <-- truncated
```

## Who hits it

Anyone whose checkout path has a space. Two shapes are common rather than exotic:

- **WSL / Windows** — `/mnt/c/Users//...`, and Windows account names with a space are the default when set up from a full name.
- **macOS** — `/Users/First Last/` is what the installer creates from a full name.

It is also unusually hard to diagnose from the error, which reads as a Ray or venv problem.

## Fix

`shlex` is already imported in `venvs.py`; the string sites just need to quote what they interpolate:

```python
-BASE = f"uv run --locked --directory {git_root}"
+BASE = f"uv run --locked --directory {shlex.quote(git_root)}"
```

across the 11 sites, which round-trips correctly through the existing `shlex.split`. Verified against all three paths above.

Structurally the cleaner fix is to carry `py_executable` as a `list[str]` and drop the split entirely, since it is only ever re-parsed — but that changes the type at every call site, so the quoting fix is the smaller change and I would rather ask which you prefer than guess.

Happy to open the PR either way — say which shape you want.

cc @yuki-97 for review or routing to the right person :)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.