[bug] Semicolon-separated Slurm params break job launch

Problem

--additional_slurm_params supports semicolon-separated values when multiple Slurm params are set, for example:

nodelist=node001,node002;reservation=my_reservation

In affected releases, setup_experiment.py consumes this flag for Slurm submission but also forwards the original argv into the rank-local run_script.py command. That command is rendered inside a bash -c '...' wrapper, so the semicolon in --additional_slurm_params is interpreted as shell syntax. The Python command is truncated at ;reservation=..., and the remaining training/config args are treated as a separate shell command, causing the job to fail immediately.

This appears to be a regression. The older r0.1.0 path built an explicit target_script_args list and did not forward --additional_slurm_params downstream.

For the upcoming LLMB release, we applied a narrow workaround that filters --additional_slurm_params out before constructing the rank-local run script: https://github.com/NVIDIA-NeMo/Megatron-Bridge/pull/3875

Longer term, setup_experiment.py should separate launcher-only args from training args, likely by splitting the parser/arg construction into setup-only, shared, and run-script args. That avoids playing whack-a-mole whenever a new launcher flag is added. A related fix might be to deprecate this flag and replace it with a repeatable one to side step the need of ';' all together. ie --slurm-arg nodelist=host[0-1] --slurm-arg reservation=blah

I’m happy to take a stab at the parser split next week.

Minimal repro

From an affected Megatron-Bridge checkout, run `setup_experiment.py` directly with dry-run enabled:


python scripts/performance/setup_experiment.py \
  --model_family_name qwen \
  --model_recipe_name qwen3_235b_a22b \
  --task pretrain \
  --compute_dtype bf16 \
  --gpu gb300 \
  --num_gpus 8 \
  --gpus_per_node 4 \
  --account dummy \
  --partition dummy \
  --log_dir /tmp/mbridge-repro \
  --container_image dummy.sqsh \
  --packager none \
  --dryrun \
  --additional_slurm_params 'nodelist=node001,node002;reservation=my_reservation' \
  train.manual_gc=true


In the generated workload command, observe that --additional_slurm_params is forwarded into the inner bash -c command unquoted.

The semicolon terminates the Python command before train.manual_gc=true.

Expected behavior

Setup specific flags shouldn't cause training to fail.

Affected area

unsure

Regression?

Yes

Environment

r0.4.0 , Nemo 26.04.00 container
r0.3.1, Nemo 26.02.01 container
(Not reproducible) r0.1.0, Nemo 25.09.00

Logs

Start with scripts/performance/setup_experiment.py and reproduce the dry run using the command in the issue. Inspect how arguments become the rank-local run_script.py command and compare that path with the older r0.1.0 behavior. Done means semicolon-separated Slurm parameters no longer truncate the inner command, while training and configuration arguments remain available.