Qwen3-235B-A22B-Instruct-2507 Lora 4机A3训练报错

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

训练脚本：

# 4机 A3设备训练
source /usr/local/Ascend/ascend-toolkit/set_env.sh
export LD_PRELOAD=/usr/local/python3.10.18/lib/python3.10/site-packages/sklearn/utils/../../scikit_learn.libs/libgomp-947d5fa1.so.1.0.0
export PYTHONPATH=$PYTHONPATH:/data/***/Qwen3-235B-A22B-Instruct-2507/test_shupei/Megatron-LM
export MEGATRON_LM_PATH=/data/***/Qwen3-235B-A22B-Instruct-2507/test_shupei/Megatron-LM

set -euo pipefail

echo "=== MS-SWIFT Multi-Node Training Started ==="
echo "NODE_RANK: ${VC_TASK_INDEX:-not_set}"
echo "MASTER_ADDR: ${MASTER_ADDR:-not_set}"
echo "PWD: $PWD"


# ============================================================================
# 1. 多机参数设置
# ============================================================================
VC_WORKER_NUM=${VC_WORKER_NUM:-1}
VC_TASK_INDEX=${VC_TASK_INDEX:-0}
export NNODES=$VC_WORKER_NUM
export NODE_RANK=$VC_TASK_INDEX

export MASTER_ADDR="127.0.0.1"
if [ "$NNODES" -gt 1 ]; then
  export MASTER_ADDR=$(awk '{print $1; exit}' /etc/volcano/worker.host)
fi

export MASTER_PORT=${MASTER_PORT:-29500}
# ============================================================================
# 2. 检查必要的环境变量
# ============================================================================
echo "检查必要的环境变量"
echo "MASTER_ADDR: ${MASTER_ADDR:-not_set}"
echo "NODE_RANK: ${NODE_RANK:-not_set}"
if [[ -z "${NODE_RANK:-}" ]] || [[ -z "${MASTER_ADDR:-}" ]]; then
  echo "ERROR: NODE_RANK or MASTER_ADDR not set"
  exit 1
fi


# 获取每个节点的 GPU 数量
NPROC_PER_NODE=16
echo "Total nodes: $NNODES"
echo "NPUs per node: $NPROC_PER_NODE"
export WORLD_SIZE=$((NNODES * NPROC_PER_NODE))
echo "WORLD_SIZE: $WORLD_SIZE"
export GLOO_SOCKET_IFNAME=eth0


# ============================================================================
# 配置日志
# ============================================================================
TIMESTAMP=$(date +'%Y%m%d_%H%M%S')
LOG_DIR="/data/***/Qwen3-235B-A22B-Instruct-2507/output/logs"

test_id=1
LOG_FILE="$LOG_DIR/tune_qwen3_235B_A22B_Instruct_2507_lora_clarify_8k_swift_${test_id}_${NODE_RANK}.log"
exec > >(tee -a "$LOG_FILE") 2>&1


# ============================================================================
# 7. 使用 torchrun 启动多机训练
# ============================================================================
echo "Starting multi-node training with torchrun..."
echo "Command: swift with torchrun"
echo "========================================"


cd /data/***/ms-swift
torchrun \
  --nnodes="$NNODES" \
  --nproc_per_node="$NPROC_PER_NODE" \
  --node_rank="$NODE_RANK" \
  --master_addr="$MASTER_ADDR" \
  --master_port=29500 \
  swift/cli/_megatron/sft.py \
  --model /data/public/models/base/Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --load_safetensors true \
  --save_safetensors true \
  --merge_lora false \
  --dataset /data/Qwen3-235B-A22B-Instruct-2507/datasets/***.jsonl \
  --val_dataset /data/Qwen3-235B-A22B-Instruct-2507/datasets/***_dev.jsonl \
  --load_from_cache_file true \
  --train_type lora \
  --lora_rank 16 \
  --lora_alpha 32 \
  --target_modules linear_qkv \
  --split_dataset_ratio 1.0 \
  --moe_permute_fusion false \
  --tensor_model_parallel_size 2 \
  --pipeline_model_parallel_size 4 \
  --expert_tensor_parallel_size 1 \
  --expert_model_parallel_size 8 \
  --moe_grouped_gemm true \
  --moe_shared_expert_overlap true \
  --moe_aux_loss_coeff 1e-4 \
  --micro_batch_size 1 \
  --global_batch_size 32 \
  --recompute_granularity full \
  --recompute_method block \
  --recompute_num_layers 12 \
  --max_epochs 2 \
  --finetune true \
  --cross_entropy_loss_fusion true \
  --lr 1e-4 \
  --lr_warmup_fraction 0.05 \
  --min_lr 1e-6 \
  --save /data/Qwen3-235B-A22B-Instruct-2507/output/ckpt/qwen3_235b_a22b_instruct_2507_clarify_TP1PP1EP8_${test_id} \
  --eval_interval 200 \
  --save_interval 200 \
  --max_length 8192 \
  --num_workers 8 \
  --dataset_num_proc 8 \
  --no_save_optim true \
  --no_save_rng true \
  --sequence_parallel true \
  --attention_backend flash \
  --model_author swift \
  --model_name swift-robot \
  --torch_dtype bfloat16 

EXIT_CODE=$?

How to Reproduce / 如何复现

版本信息：

环境信息收集报告

收集时间: 2026-01-29 15:46:20

【系统信息】

操作系统:

Linux

系统版本:

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

系统架构:

aarch64

Python版本:

3.10.18

Python路径:

/usr/local/python3.10.18/bin/python3.10

================================================================================

【硬件信息】

npu_info:

+------------------------------------------------------------------------------------------------+
| npu-smi 25.3.rc1.2 Version: 25.3.rc1.2 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip Phy-ID | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 4 Ascend910 | OK | 160.2 35 0 / 0 |
| 0 8 | 0000:8D:00.0 | 0 0 / 0 3149 / 65536 |
+------------------------------------------------------------------------------------------------+
| 4 Ascend910 | OK | - 35 0 / 0 |
| 1 9 | 0000:8F:00.0 | 0 0 / 0 2885 / 65536 |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===========================+===============+====================================================+
| No running processes found in NPU 4 |
+===========================+===============+====================================================+

npu_version:

npu-smi version: 25.3.rc1.2

cpu_info:

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 640
On-line CPU(s) list: 0-639
Thread(s) per core: 2
Core(s) per socket: 80
Socket(s): 4
NUMA node(s): 8
Vendor ID: 0x48
Model: 0
Stepping: 0x0
Frequency boost: disabled
CPU max MHz: 2900.0000
CPU min MHz: 400.0000
BogoMIPS: 200.00
L1d cache: 20 MiB
L1i cache: 20 MiB
L2 cache: 400 MiB
L3 cache: 560 MiB
NUMA node0 CPU(s): 0-79
NUMA node1 CPU(s): 80-159
NUMA node2 CPU(s): 160-239
NUMA node3 CPU(s): 240-319
NUMA node4 CPU(s): 320-399
NUMA node5 CPU(s): 400-479
NUMA node6 CPU(s): 480-559
NUMA node7 CPU(s): 560-639
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb dcpodp flagm2 frint svei8mm svef32mm svef64mm svebf16 i8mm bf16 dgh rng ecv

memory_info:

          total        used        free      shared  buff/cache   available

Mem: 2.0Ti 356Gi 1.3Ti 4.1Gi 334Gi 1.6Ti
Swap: 0B 0B 0B

cann_version:

version: 1.0

runtime_running_version=[8.2.0.0.201:8.2.RC1]
compiler_running_version=[8.2.0.0.201:8.2.RC1]
hccl_running_version=[8.2.0.0.201:8.2.RC1]
opp_running_version=[8.2.0.0.201:8.2.RC1]
toolkit_running_version=[8.2.0.0.201:8.2.RC1]
aoe_running_version=[8.2.0.0.201:8.2.RC1]
ncs_running_version=[8.2.0.0.201:8.2.RC1]
opp_kernel_running_version=[8.2.0.0.201:8.2.RC1]
runtime_upgrade_version=[8.2.0.0.201:8.2.RC1]
compiler_upgrade_version=[8.2.0.0.201:8.2.RC1]
hccl_upgrade_version=[8.2.0.0.201:8.2.RC1]
opp_upgrade_version=[8.2.0.0.201:8.2.RC1]
toolkit_upgrade_version=[8.2.0.0.201:8.2.RC1]
aoe_upgrade_version=[8.2.0.0.201:8.2.RC1]
ncs_upgrade_version=[8.2.0.0.201:8.2.RC1]
opp_kernel_upgrade_version=[8.2.0.0.201:8.2.RC1]
runtime_installed_version=[8.2.0.0.201:8.2.RC1]
compiler_installed_version=[8.2.0.0.201:8.2.RC1]
hccl_installed_version=[8.2.0.0.201:8.2.RC1]
opp_installed_version=[8.2.0.0.201:8.2.RC1]
toolkit_installed_version=[8.2.0.0.201:8.2.RC1]
aoe_installed_version=[8.2.0.0.201:8.2.RC1]
ncs_installed_version=[8.2.0.0.201:8.2.RC1]
opp_kernel_installed_version=[8.2.0.0.201:8.2.RC1]

gpu_info:

未找到信息

cuda_version:

未找到信息

================================================================================

【框架信息】

基础库:

PyTorch:
版本: 2.7.1
关键依赖:
- numpy: 1.24.0
- tqdm: 4.67.1
PyTorch_NPU:
版本: 2.7.1
关键依赖:
- torch: 2.7.1
Transformers:
版本: 4.57.6
关键依赖:
- torch: 2.7.1
- tokenizers: 0.22.2
- huggingface-hub: 0.36.0
Datasets:
版本: 3.6.0
关键依赖:
- torch: 2.7.1
- numpy: 1.24.0
- pandas: 2.3.3
Accelerate:
版本: 1.12.0
关键依赖:
- torch: 2.7.1
- numpy: 1.24.0

训练框架:
Swift:
版本: 3.12.2
关键依赖:
- torch: 2.7.1
- torch_npu: 2.7.1
- transformers: 4.57.6
- peft: 0.18.1
- mindspeed: 0.12.1
- datasets: 3.6.0
- accelerate: 1.12.0

Additional Information / 补充信息

No response

Start with the reported torchrun command and swift/cli/megatron/sft.py, then inspect the generated log at output/logs/tune_qwen3_235B_A22B-Instruct-2507_lora_clarify_8k_swift${test_id}_${NODE_RANK}.log. Reproduce the four-node Ascend 910 LoRA run using the listed Swift, PyTorch, PyTorch-NPU, and CANN versions, identify the reported failure, and confirm that training completes without that error.