modelscope / modelscope/ms-swift

华为910B卡上,GRPO训练推理乱码问题

Open
#6,841 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

npu stale
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Describe the bug
grpo训练按如下方式启动grpo训练:
(1)ASCEND_RT_VISIBLE_DEVICES=5 swift rollout --model /yunpan/zhaoyin/model/Qwen2___5-1___5B-Instruct/ --tensor_parallel_size 1
(2)ASCEND_RT_VISIBLE_DEVICES=3,4 NPROC_PER_NODE=2 swift rlhf --rlhf_type grpo --model /yunpan/zhaoyin/model/Qwen2___5-1___5B-Instruct/ --reward_funcs string_sim --use_vllm true --vllm_mode server --vllm_server_host 127.0.0.1 --vllm_server_port 8000 --train_type full --torch_dtype bfloat16 --dataset /yunpan/zy_exp/other/LW97.LT23054_llm_trainer/train_test/gsm8k_data/train.jsonl --split_dataset_ratio 0 --save_steps 200 --max_completion_length 4096 --max_length 512 --num_train_epochs 1 --per_device_train_batch_size 1 --learning_rate 2e-5 --gradient_accumulation_steps 1 --logging_steps 1 --warmup_ratio 0.05 --dataloader_num_workers 4 --dataset_num_proc 4 --num_generations 2 --temperature 0.3 --deepspeed zero2_offload --log_completions true --num_iterations 1 --beta 0.001
训练过程中推理结果全是乱的,疑似update_flattened_param方法从训练端传给server端的权重是空的。还原为原本的update_named_param方法后结果正常。

Image

Your hardware and system info
Ascend 910B3, torch/torch_npu=2.7.1, vllm/vllm_ascend=0.11.0

Additional context
Add any other context about the problem here(在这里补充其他信息)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing update_flattened_param with update_named_param in the training-to-server weight transfer path. Reproduce the GRPO commands on Ascend 910B hardware and verify whether the server receives nonempty, valid weights and produces normal outputs. Done means the flattened-parameter path works without restoring the named-parameter implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.