Qwen3.5-397B-A17B 全异步 H20（96 Actor，64 Rollout）运行正常，但 Rollout 生成的 Response 不正常

Your Question

在 32 张 H20 上以全异步模式训练 Qwen3.5-35B-A3B 时一切正常。采用完全相同的环境，扩展至 160 张 H20（96 张负责训练，64 张负责 Rollout）训练 Qwen3.5-397B-A17B，训练流程可以正常运行，但出现 Rollout 输出乱码的问题。请问可能是什么原因导致的？

hf 2 megatron

35B 和 397B 均通过 /tools/convert_torch_dist_to_hf.py 脚本进行权重转换，其中 35B 采用单机转换，397B 采用四机转换。两个模型在转换过程中均未报任何错误。

但是该脚本在转换时会自动去除 Visual 部分的参数（即仅保留语言模型权重）。以 35B 为例，转换完成后得到的 megatron 通过 /tools/convert_torch_dist_to_hf_parallel.py 转为 HF 格式权重，可通过 vLLM 正常加载推理，推理时需将相关参数设置为 Language Model Only 模式，推理结果正常。397b转换过程同样会去除visual 参数，目前不太清楚是不是和模型转换有关。

训练脚本参数配置

===== 训练数据 =====
DAPO-math-17k

===== Checkpoint 参数 =====
CKPT_ARGS = [
"--hf-checkpoint", PRETRAIN_MODEL,
"--ref-load", PRETRAIN_MODEL_CKPT,
"--load", CKPT_LOAD_PATH,
"--save", CKPT_SAVE_PATH,
"--save-interval", "20",
]

===== 动态生成 moe_layer_freq =====
NLAYERS = 60
FIRST_K_DENSE_REPLACE = 0
moe_layer_freq_list = [0 if i < FIRST_K_DENSE_REPLACE else 1 for i in range(NLAYERS)]
MOE_LAYER_FREQ = "[" + ",".join(str(x) for x in moe_layer_freq_list) + "]"

===== Qwen3.5-397B-A17B 模型结构参数 =====
MODEL_ARGS = [
"--spec", "slime_plugins.models.qwen3_5", "get_qwen3_5_spec",
"--disable-bias-linear",
"--qk-layernorm",
"--group-query-attention",
"--num-attention-heads", "32", # num_attention_heads=32
"--num-query-groups", "2", # num_key_value_heads=2（GQA）
"--kv-channels", "256", # head_dim=256
"--num-layers", "60", # num_hidden_layers=60
"--hidden-size", "4096", # hidden_size=4096
"--ffn-hidden-size", "1024", # shared_expert_intermediate_size=1024（dense FFN 对齐）
"--use-gated-attention",
"--normalization", "RMSNorm",
"--apply-layernorm-1p",
"--position-embedding-type", "rope",
"--norm-epsilon", "1e-6", # rms_norm_eps=1e-06
"--rotary-percent", "0.25", # partial_rotary_factor=0.25
"--swiglu",
"--untie-embeddings-and-output-weights",
"--vocab-size", "248320",
"--rotary-base", "10000000", # rope_theta=10000000
"--moe-ffn-hidden-size", "1024", # moe_intermediate_size=1024
"--moe-shared-expert-intermediate-size", "1024", # shared_expert_intermediate_size=1024
"--moe-router-score-function", "softmax",
"--moe-token-dispatcher-type", "alltoall",
"--moe-router-topk", "10", # num_experts_per_tok=10
"--moe-layer-freq", MOE_LAYER_FREQ,
"--num-experts", "512", # num_experts=512
"--moe-grouped-gemm",
"--moe-token-drop-policy", "probs",
"--moe-router-dtype", "fp32",
"--moe-permute-fusion",
"--moe-aux-loss-coeff", "0",
"--attention-output-gate",
"--moe-shared-expert-gate",
]

===== Rollout 参数 =====
ROLLOUT_ARGS = [
"--rollout-function-path", "examples.fully_async.fully_async_rollout.generate_rollout_fully_async",
"--prompt-data", "$TRAIN_DATA_PATH",
"--input-key", "prompt",
"--label-key", "label",
"--apply-chat-template",
"--rollout-shuffle",
"--rm-type", "deepscaler",
"--num-epoch", "1",
"--rollout-batch-size", "64",
"--n-samples-per-prompt", "4",
"--rollout-max-response-len", "16384",
"--rollout-temperature", "1.0",
"--global-batch-size", "128",
"--balance-data",
"--rollout-global-dataset",
"--sglang-server-concurrency", "96", # 4 engines × 16 GPU
]

===== 并行策略参数 =====
PERF_ARGS = [
"--tensor-model-parallel-size", "2", # hidden=4096/2=2048/卡，KV heads=2/2=1/卡 ✅
"--sequence-parallel",
"--pipeline-model-parallel-size", "6", # 60层 / 6 = 每 stage 10 层
"--context-parallel-size", "1", # CP=1
"--expert-model-parallel-size", "8", # 512 experts / 8 = 64 experts/GPU ✅
"--expert-tensor-parallel-size", "1",
"--recompute-granularity", "full",
"--recompute-method", "uniform",
"--recompute-num-layers", "2", # 每 stage 10 层，重计算 2 层
"--use-dynamic-batch-size",
"--calculate-per-token-loss",
"--max-tokens-per-gpu", "4096", # 397B 显存更紧张，从 8192 减半
]

===== GRPO 参数 =====
GRPO_ARGS = [
"--advantage-estimator", "grpo",
"--kl-loss-coef", "0.00",
"--kl-loss-type", "low_var_kl",
"--kl-coef", "0.00",
"--entropy-coef", "0.00",
"--eps-clip", "0.2",
]

===== 优化器参数 =====
OPTIMIZER_ARGS = [
"--optimizer", "adam",
"--lr", "1e-6",
"--lr-decay-style", "constant",
"--weight-decay", "0.1",
"--adam-beta1", "0.9",
"--adam-beta2", "0.98",
"--optimizer-cpu-offload",
"--overlap-cpu-optimizer-d2h-h2d",
"--use-precision-aware-optimizer",
]

===== SGLang 参数 =====

SGLANG_ARGS = [
"--rollout-num-gpus", "64", # rollout 侧总 GPU 数（4 engines × 16 GPU）
"--rollout-num-gpus-per-engine", "16", # 每个 engine 16 GPU（TP=16）
"--sglang-mem-fraction-static", "0.7",
"--sglang-speculative-algorithm", "EAGLE",
"--sglang-speculative-num-steps", "3",
"--sglang-speculative-eagle-topk", "1",
"--sglang-speculative-num-draft-tokens", "4",
]

===== 其他参数 =====
MISC_ARGS = [
"--attention-dropout", "0.0",
"--hidden-dropout", "0.0",
"--accumulate-allreduce-grads-in-fp32",
"--attention-softmax-in-fp32",
"--attention-backend", "flash",
]

===== 拼接训练命令 =====
def build_command(parts):
"""对含 shell 特殊字符（,）的参数加双引号，防止 shell 展开或拆分"""
result = []
for p in parts:
if any(c in p for c in ','):
result.append(f'"{p}"')
else:
result.append(p)
return " ".join(result)

train_cmd_parts = (
["python3", "train_async.py"] +
# 训练侧 12 节点 96 卡
["--actor-num-nodes", "12"] + # 训练侧 96 卡 = 12 节点 × 8 卡
["--actor-num-gpus-per-node", "8"] +
MODEL_ARGS +
CKPT_ARGS +
ROLLOUT_ARGS +
OPTIMIZER_ARGS +
GRPO_ARGS +
PERF_ARGS +
SGLANG_ARGS +
WANDB_ARGS +
MISC_ARGS
)

qwen3.5-397B-A17B.sh

NLAYERS=60
FIRST_K_DENSE_REPLACE=0

arr=()
for ((i=0; i<NLAYERS; i++)); do
if (( i < FIRST_K_DENSE_REPLACE )); then
arr+=(0)
else
arr+=(1)
fi
done

printf -v MOE_LAYER_FREQ "[%s]" "$(IFS=', '; echo "${arr[*]}")"

MODEL_ARGS=(
--spec "slime_plugins.models.qwen3_5" "get_qwen3_5_spec"

--disable-bias-linear
--qk-layernorm
--group-query-attention
--num-attention-heads 32
--num-query-groups 2
--kv-channels 256
--num-layers 60
--hidden-size 4096
--ffn-hidden-size 1024
--use-gated-attention

--normalization RMSNorm
--apply-layernorm-1p
--position-embedding-type rope
--norm-epsilon 1e-6
--rotary-percent 0.25
--swiglu
--untie-embeddings-and-output-weights
--vocab-size 248320

--rotary-base 10000000

--moe-ffn-hidden-size 1024
--moe-shared-expert-intermediate-size 1024
--moe-router-score-function softmax
--moe-token-dispatcher-type alltoall
--moe-router-topk 10
--moe-layer-freq "$MOE_LAYER_FREQ"
--num-experts 512
--moe-grouped-gemm
--moe-token-drop-policy probs
--moe-router-dtype fp32
--moe-permute-fusion
--moe-aux-loss-coeff 0

--attention-output-gate
--moe-shared-expert-gate
)

397b 四机转换脚本

!/bin/bash

===================== NCCL =====================
export NCCL_IB_GID_INDEX=3
export NCCL_SOCKET_NTHREADS=8
export NCCL_IB_TIMEOUT=22
export NCCL_ASYNC_ERROR_HANDLING=1
export NCCL_P2P_LEVEL=NVL

===================== 路径 =====================
hf_checkpoint=""
save_path=""
megatron_lm_path="/root/Megatron-LM"
slime_path=""

NPROC_PER_NODE=8

NODE_RANK=${RANK} # RANK=节点rank (0,1,2,3)
NNODES=${WORLD_SIZE} # WORLD_SIZE=节点总数 (4)

echo "===== torchrun 参数 ====="
echo " NODE_RANK = ${NODE_RANK}"
echo " NNODES = ${NNODES}"
echo " MASTER_ADDR = ${MASTER_ADDR}"
echo " MASTER_PORT = ${MASTER_PORT}"
echo " NPROC_PER_NODE = ${NPROC_PER_NODE}"
echo "========================="

===================== 网络接口 =====================
DETECTED_IF=$(ip route get "${MASTER_ADDR}" 2>/dev/null
| awk '/dev/{for(i=1;i<=NF;i++) if($i=="dev") print $(i+1)}' | head -1)
export NCCL_SOCKET_IFNAME="${DETECTED_IF:-eth0}"
echo "NCCL_SOCKET_IFNAME=${NCCL_SOCKET_IFNAME}"

===================== 执行 =====================
cd "${slime_path}"
source scripts/models/qwen3.5-397B-A17B.sh

PYTHONPATH="${megatron_lm_path}" torchrun
--nproc-per-node "${NPROC_PER_NODE}"
--nnodes "${NNODES}"
--node_rank "${NODE_RANK}"
--master_addr "${MASTER_ADDR}"
--master_port "${MASTER_PORT}"
tools/convert_hf_to_torch_dist.py
"${MODEL_ARGS[@]}"
--pipeline-model-parallel-size 2
--expert-model-parallel-size 16
--hf-checkpoint "${hf_checkpoint}"
--save "${save_path}"

What I've Tried

尝试更新sglang，但是更新之后运行环境报错，导致训练无法进行。
目前考虑是不是模型转换过程的bug。

Environment (if relevant)

• slime version: 0.2.4
• Python version:
• PyTorch version: 2.9.1+cu129
• CUDA/ROCm version:
• GPU type and count: h20
• OS:
  transformers 5.3.0
  megatron-core 0.16.0rc0
  sglang 0.5.9
  mbridge 0.15.1
  megatron-bridge 0.3.0rc0

Additional Context

No response

Pre-submission Checklist

• I have read the CONTRIBUTING.md and understand the collaboration scope.
• I have read the documentation and FAQ and my question is not answered there.
• I have searched for existing issues and my question has not been asked before.

Start with tools/convert_torch_dist_to_hf.py, tools/convert_torch_dist_to_hf_parallel.py, and the fully asynchronous rollout entry point examples.fully_async.fully_async_rollout.generate_rollout_fully_async. Reproduce the 35B and 397B conversion and rollout paths, then compare the converted checkpoints and Language Model Only settings used with vLLM. Done means identifying whether conversion or rollout configuration produces the malformed responses and documenting reproducible evidence.