modelscope / modelscope/ms-swift
华为910B卡上,GRPO训练推理乱码问题
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Describe the bug
grpo训练按如下方式启动grpo训练:
(1)ASCEND_RT_VISIBLE_DEVICES=5 swift rollout --model /yunpan/zhaoyin/model/Qwen2___5-1___5B-Instruct/ --tensor_parallel_size 1
(2)ASCEND_RT_VISIBLE_DEVICES=3,4 NPROC_PER_NODE=2 swift rlhf --rlhf_type grpo --model /yunpan/zhaoyin/model/Qwen2___5-1___5B-Instruct/ --reward_funcs string_sim --use_vllm true --vllm_mode server --vllm_server_host 127.0.0.1 --vllm_server_port 8000 --train_type full --torch_dtype bfloat16 --dataset /yunpan/zy_exp/other/LW97.LT23054_llm_trainer/train_test/gsm8k_data/train.jsonl --split_dataset_ratio 0 --save_steps 200 --max_completion_length 4096 --max_length 512 --num_train_epochs 1 --per_device_train_batch_size 1 --learning_rate 2e-5 --gradient_accumulation_steps 1 --logging_steps 1 --warmup_ratio 0.05 --dataloader_num_workers 4 --dataset_num_proc 4 --num_generations 2 --temperature 0.3 --deepspeed zero2_offload --log_completions true --num_iterations 1 --beta 0.001
训练过程中推理结果全是乱的,疑似update_flattened_param方法从训练端传给server端的权重是空的。还原为原本的update_named_param方法后结果正常。
Your hardware and system info
Ascend 910B3, torch/torch_npu=2.7.1, vllm/vllm_ascend=0.11.0
Additional context
Add any other context about the problem here(在这里补充其他信息)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing update_flattened_param with update_named_param in the training-to-server weight transfer path. Reproduce the GRPO commands on Ascend 910B hardware and verify whether the server receives nonempty, valid weights and produces normal outputs. Done means the flattened-parameter path works without restoring the named-parameter implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100