Unable to get Qwen3-0.6B running

This might be a silly question, but I figured I would still ask it.

I followed similar steps to the quick_start to try and get the Qwen3-0.6B model running. But I am unable to run anything with it.

To convert the HF to Megatron format, I ran: PYTHONPATH=/root/Megatron-LM p^Chon tools/convert_hf_to_torch_dist.py ${MODEL_ARGS[@]} --hf-checkpoint /root/qwen3-0.6B --save /root/qwen3-0.6B_torch_dist.

Then I ran source scripts/models/qwen3-0.6B.sh.

Finally, I ran bash scripts/run-qwen3-06B.sh.

I get the following stack trace:

Stopped all 147 Ray processes.
+ export PYTHONBUFFERED=16
+ PYTHONBUFFERED=16
+ export CUDA_VISIBLE_DEVICES=2,5
+ CUDA_VISIBLE_DEVICES=2,5
+ echo 'Starting the modified script'
Starting the modified script
+ TRAINING_GPU=1
+ INFERENCE_GPU=1
+ TOTAL_GPUS=2
++ nvidia-smi topo -m
++ grep -o 'NV[0-9][0-9]*'
++ wc -l
+ NVLINK_COUNT=56
+ '[' 56 -gt 0 ']'
+ HAS_NVLINK=1
+ echo 'HAS_NVLINK: 1 (detected 56 NVLink references)'
HAS_NVLINK: 1 (detected 56 NVLink references)
+++ dirname -- scripts/run-qwen3-06B-copy.sh
++ cd -- scripts
++ pwd
+ SCRIPT_DIR=/root/slime/scripts
+ source /root/slime/scripts/models/qwen3-0.6B.sh
++ MODEL_ARGS=(--swiglu --num-layers 28 --hidden-size 1024 --ffn-hidden-size 3072 --num-attention-heads 16 --group-query-attention --num-query-groups 8 --use-rotary-position-embeddings --disable-bias-linear --normalization "RMSNorm" --norm-epsilon 1e-6 --rotary-base 1000000 --vocab-size 151936 --kv-channels 128 --qk-layernorm)
+ CKPT_ARGS=(--hf-checkpoint /root/Qwen3-0.6B --ref-load /root/Qwen3-0.6B_torch_dist --load /root/Qwen3-0.6B_slime/ --save /root/Qwen3-0.6B_slime/ --save-interval 20)
+ ROLLOUT_ARGS=(--prompt-data /root/dapo-math-17k/dapo-math-17k.jsonl --input-key prompt --label-key label --apply-chat-template --rollout-shuffle --rm-type deepscaler --num-rollout 2 --rollout-batch-size 32 --n-samples-per-prompt 8 --rollout-max-response-len 32768 --rollout-temperature 0.8 --global-batch-size 256 --balance-data)
+ EVAL_ARGS=(--eval-interval 20 --eval-prompt-data aime /root/aime-2024/aime-2024.jsonl --n-samples-per-eval-prompt 16 --eval-max-response-len 16384 --eval-top-p 0.7)
+ PERF_ARGS=(--tensor-model-parallel-size 1 --sequence-parallel --pipeline-model-parallel-size 1 --context-parallel-size 1 --expert-model-parallel-size 1 --expert-tensor-parallel-size 1 --recompute-granularity full --recompute-method uniform --recompute-num-layers 1 --use-dynamic-batch-size --max-tokens-per-gpu 9216)
+ GRPO_ARGS=(--advantage-estimator grpo --use-kl-loss --kl-loss-coef 0.00 --kl-loss-type low_var_kl --entropy-coef 0.00 --eps-clip 0.2 --eps-clip-high 0.28)
+ OPTIMIZER_ARGS=(--optimizer adam --lr 1e-6 --lr-decay-style constant --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98)
+ WANDB_ARGS=()
+ SGLANG_ARGS=(--rollout-num-gpus-per-engine 1 --sglang-mem-fraction-static 0.85)
+ MISC_ARGS=(--attention-dropout 0.0 --hidden-dropout 0.0 --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 --attention-backend flash)
+ export MASTER_ADDR=127.0.0.1
+ MASTER_ADDR=127.0.0.1
+ ray start --head --node-ip-address 127.0.0.1 --num-gpus 2 --disable-usage-stats --dashboard-host=0.0.0.0 --dashboard-port=8265 --temp-dir /root/slime/logs
Usage stats collection is disabled.

Local node IP: 172.17.0.9
2025-11-19 06:23:28,509 WARNING utils.py:460 -- Detecting docker specified CPUs. In previous versions of Ray, CPU detection in containers was incorrect. Please ensure that Ray has enough CPUs allocated. As a temporary workaround to revert to the prior behavior, set `RAY_USE_MULTIPROCESSING_CPU_COUNT=1` as an env var before starting Ray. Set the env var: `RAY_DISABLE_DOCKER_CPU_WARNING=1` to mute this warning.
2025-11-19 06:23:28,510 WARNING utils.py:472 -- Ray currently does not support initializing Ray with fractional cpus. Your num_cpus will be truncated from 230.4 to 230.

--------------------
Ray runtime started.
--------------------

Next steps
  To add another node to this Ray cluster, run
    ray start --address='172.17.0.9:6379'

  To connect to this Ray cluster:
    import ray
    ray.init(_node_ip_address='172.17.0.9')

  To submit a Ray job using the Ray Jobs CLI:
    RAY_API_SERVER_ADDRESS='http://172.17.0.9:8265' ray job submit --working-dir . -- python my_script.py

  See https://docs.ray.io/en/latest/cluster/running-applications/job-submission/index.html
  for more information on submitting Ray jobs to the Ray cluster.

  To terminate the Ray runtime, run
    ray stop

  To view the status of the cluster, use
    ray status

  To monitor and debug Ray, view the dashboard at
    172.17.0.9:8265

  If connection to the dashboard fails, check your firewall settings and network configuration.
+ RUNTIME_ENV_JSON='{
  "env_vars": {
    "PYTHONPATH": "/root/Megatron-LM/",
    "CUDA_DEVICE_MAX_CONNECTIONS": "1",
    "NCCL_NVLS_ENABLE": "1"
  }
}'
+ ray job submit --address=http://127.0.0.1:8265 '--runtime-env-json={
  "env_vars": {
    "PYTHONPATH": "/root/Megatron-LM/",
    "CUDA_DEVICE_MAX_CONNECTIONS": "1",
    "NCCL_NVLS_ENABLE": "1"
  }
}' -- python3 train_async.py --actor-num-nodes 1 --actor-num-gpus-per-node 1 --rollout-num-gpus 1 --swiglu --num-layers 28 --hidden-size 1024 --ffn-hidden-size 3072 --num-attention-heads 16 --group-query-attention --num-query-groups 8 --use-rotary-position-embeddings --disable-bias-linear --normalization RMSNorm --norm-epsilon 1e-6 --rotary-base 1000000 --vocab-size 151936 --kv-channels 128 --qk-layernorm --hf-checkpoint /root/Qwen3-0.6B --ref-load /root/Qwen3-0.6B_torch_dist --load /root/Qwen3-0.6B_slime/ --save /root/Qwen3-0.6B_slime/ --save-interval 20 --prompt-data /root/dapo-math-17k/dapo-math-17k.jsonl --input-key prompt --label-key label --apply-chat-template --rollout-shuffle --rm-type deepscaler --num-rollout 2 --rollout-batch-size 32 --n-samples-per-prompt 8 --rollout-max-response-len 32768 --rollout-temperature 0.8 --global-batch-size 256 --balance-data --optimizer adam --lr 1e-6 --lr-decay-style constant --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.98 --advantage-estimator grpo --use-kl-loss --kl-loss-coef 0.00 --kl-loss-type low_var_kl --entropy-coef 0.00 --eps-clip 0.2 --eps-clip-high 0.28 --tensor-model-parallel-size 1 --sequence-parallel --pipeline-model-parallel-size 1 --context-parallel-size 1 --expert-model-parallel-size 1 --expert-tensor-parallel-size 1 --recompute-granularity full --recompute-method uniform --recompute-num-layers 1 --use-dynamic-batch-size --max-tokens-per-gpu 9216 --eval-interval 20 --eval-prompt-data aime /root/aime-2024/aime-2024.jsonl --n-samples-per-eval-prompt 16 --eval-max-response-len 16384 --eval-top-p 0.7 --rollout-num-gpus-per-engine 1 --sglang-mem-fraction-static 0.85 --attention-dropout 0.0 --hidden-dropout 0.0 --accumulate-allreduce-grads-in-fp32 --attention-softmax-in-fp32 --attention-backend flash
Job submission server address: http://127.0.0.1:8265

-------------------------------------------------------
Job 'raysubmit_pk9BETsLBSMughPE' submitted successfully
-------------------------------------------------------

Next steps
  Query the logs of the job:
    ray job logs raysubmit_pk9BETsLBSMughPE
  Query the status of the job:
    ray job status raysubmit_pk9BETsLBSMughPE
  Request the job to be stopped:
    ray job stop raysubmit_pk9BETsLBSMughPE

Tailing logs until the job exits (disable with --no-wait):
2025-11-19 06:23:37,005 INFO job_manager.py:568 -- Runtime env is setting up.
2025-11-19 06:23:52 - INFO - opentelemetry package is not installed, tracing disabled
2025-11-19 06:23:57 - INFO - Detected Megatron Core, using Megatron-FSDP with Megatron.
2025-11-19 06:23:57 - INFO - Detected Megatron Core, using Megatron-FSDP with Megatron.
Traceback (most recent call last):
  File "/root/slime/train_async.py", line 83, in <module>
    args = parse_args()
           ^^^^^^^^^^^^
  File "/root/slime/slime/utils/arguments.py", line 1133, in parse_args
    hf_validate_args(args, hf_config)
  File "/root/slime/slime/utils/arguments.py", line 1398, in hf_validate_args
    assert compare_fn(getattr(hf_config, hf_config_name), getattr(args, megatron_config_name)), (
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: hidden_size in hf config 896 is not equal to hidden_size 1024, please check the config.

I am assuming the config is right because I was able to run Qwen3 0.6B a few weeks ago. Is there anything I'm doing run / forgetting to do? Thanks!

Start by reproducing the command sequence from scripts/models/qwen3-0.6B.sh and scripts/run-qwen3-06B.sh, then read parse_args and hf_validate_args in slime/utils/arguments.py, reached from train_async.py. Compare the model configuration with the generated arguments and verify that the Qwen3 quick-start run completes without the hidden_size assertion.