NVIDIA-NeMo / NVIDIA-NeMo/Automodel
--nproc-per-node is overridden by the local GPU count
@HuiyingLi is already working on this.
Since Sep 1, 2026.
- Dominant language
- Python
- Stars
- 960
- Forks
- 316
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
Describe the bug
InteractiveLauncher.launch consults the local GPU count before it honors an
explicit --nproc-per-node, so the flag is overridden by the device probe in
two ways.
nemo_automodel/components/launcher/interactive.py:
nproc_per_node: int | None = launcher_config
...
num_devices = determine_local_world_size(nproc_per_node="gpu")
assert num_devices > 0, "Expected num-devices to be > 0"
if nproc_per_node == 1 or num_devices == 1:
logger.info("Launching job locally on a single device")
return self._run_recipe_in_process(recipe_target, config)
else:
effective_nproc = nproc_per_node if nproc_per_node is not None else num_devices
1. --nproc-per-node N is silently downgraded to a single process on a
one-GPU host.
num_devices == 1 short-circuits to the in-process path regardless of what the
user asked for, so on a single-GPU box:
automodel config.yaml --nproc-per-node 4
runs one process, not four, and logs "Launching job locally on a single
device" without mentioning that the requested value was discarded. Running
several ranks on one device is a normal way to exercise FSDP/DDP or
pipeline-parallel code paths on a dev box, and torchrun --nproc-per-node 4
supports it directly.
2. --nproc-per-node 1 fails on a host with no visible CUDA device.
determine_local_world_size(nproc_per_node="gpu") raises when
torch.cuda.is_available() is false, and it runs before the
nproc_per_node == 1 branch — the one path that needs no device count at all,
because it runs the recipe in-process:
$ automodel config.yaml --nproc-per-node 1
...
File ".../torch/distributed/run.py", line 745, in determine_local_world_size
raise ValueError("Cuda is not available.") from e
ValueError: Cuda is not available.
This also fires when CUDA_VISIBLE_DEVICES="" is set. The error surfaces from
inside torch.distributed.run, so it does not say which AutoModel flag or
requirement is involved.
Steps/Code to reproduce bug
Symptom 2 reproduces on any host without a visible CUDA device:
automodel examples/llm_finetune/llama3_2/llama3_2_1b_squad.yaml --nproc-per-node 1
Symptom 1 reproduces on a single-GPU host with --nproc-per-node 4, or in a
unit test by stubbing determine_local_world_size to return 1: the launcher
takes the in-process branch and torch.distributed.run.run is never called.
Expected behavior
An explicit --nproc-per-node N is honored: 1 runs in-process, N > 1
launches torchrun with N workers. The GPU count is only probed when the flag
is not supplied, since that is the only case where it is needed to pick a
default.
--help describes the flag as "Number of workers per node for local/interactive
jobs", with no mention of it being clamped to the visible device count.
Environment overview
mainat 3ddef9b1. Symptom 2 observed on a CPU-only host; symptom 1 is
reachable in a CPU unit test by stubbing the probe.
Additional context
To be clear about scope: fixing symptom 2 does not make CPU training work —
the recipe still requires CUDA further down. It makes the documented flag
behave as documented and moves the failure to where the framework can report
its own requirement, instead of failing in a device probe that the requested
single-process path never needed.
Happy to send a PR: probe only when nproc_per_node is None, plus CPU unit
tests for both symptoms.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.