modelscope / modelscope/ms-swift
gkd colocate 脚本 eval时出错
Open
Nobody has claimed this yet.
stale
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Describe the bug
What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程，最好有截图)
运行gkd/vllm_colocate.sh 脚本时，能够正常训练，直到eval steps发生下述报错
这里是报错：
INFO 11-26 22:39:18 [executor_base.py:205] It took 0.161757 seconds to wake up tags ['weights'].
INFO 11-26 22:39:18 [executor_base.py:205] It took 0.162088 seconds to wake up tags ['weights'].
INFO 11-26 22:39:18 [executor_base.py:205] It took 0.162590 seconds to wake up tags ['weights'].
INFO 11-26 22:39:18 [executor_base.py:205] It took 0.163555 seconds to wake up tags ['weights'].
[INFO:swift] last_model_checkpoint: None
[INFO:swift] best_model_checkpoint: None
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 465, in _move_full_model_to_vllm
self._load_state_dict_to_vllm(state_dict)
File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 421, in _load_state_dict_to_vllm
llm_model.load_weights(state_dict.items())
File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/qwen3.py", line 341, in load_weights
return loader.load_weights(weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 294, in load_weights
autoloaded_weights = set(self._load_module("", self.module, weights))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: Traceback (most recent call last):
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 465, in _move_full_model_to_vllm
[rank1]: self._load_state_dict_to_vllm(state_dict)
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 421, in _load_state_dict_to_vllm
[rank1]: llm_model.load_weights(state_dict.items())
[rank1]: File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/qwen3.py", line 341, in load_weights
[rank1]: return loader.load_weights(weights)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 294, in load_weights
[rank1]: autoloaded_weights = set(self._load_module("", self.module, weights))
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 252, in _load_module
[rank1]: yield from self._load_module(prefix,
[rank1]: File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 225, in _load_module
[rank1]: loaded_params = module_load_weights(weights)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 425, in load_weights
[rank1]: weight_loader(param, loaded_weight, shard_id)
[rank1]: File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/layers/linear.py", line 1016, in weight_loader_v2
[rank1]: param.load_qkv_weight(loaded_weight=loaded_weight,
[rank1]: File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/parameter.py", line 192, in load_qkv_weight
[rank1]: loaded_weight = loaded_weight.narrow(self.output_dim,
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: RuntimeError: start (0) + length (1024) exceeds dimension size (0).
rank1]: Traceback (most recent call last):
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/cli/rlhf.py", line 7, in <module>
[rank1]: rlhf_main()
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank1]: return SwiftRLHF(args).main()
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/llm/base.py", line 49, in main
[rank1]: result = self.run()
[rank1]: ^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/ray/base.py", line 170, in wrapper
[rank1]: return func(self, *args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/sft.py", line 207, in run
[rank1]: return self.train(trainer)
[rank1]: ^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/sft.py", line 255, in train
[rank1]: trainer.train(trainer.args.resume_from_checkpoint)
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/mixin.py", line 815, in train
[rank1]: res = super().train(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2325, in train
[rank1]: return inner_training_loop(
[rank1]: ^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank1]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 407, in wrapper
[rank1]: return func(self, *args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/gkd_trainer.py", line 202, in training_step
[rank1]: generated_inputs = self._fast_infer(processed_inputs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 643, in _fast_infer
[rank1]: self._move_model_to_vllm()
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 407, in wrapper
[rank1]: return func(self, *args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 357, in _move_model_to_vllm
[rank1]: self._move_full_model_to_vllm()
[rank1]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 438, in _move_full_model_to_vllm
[rank1]: with gather_if_zero3(parameters), context_manager:
[rank1]: File "/usr/local/lib/python3.11/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 2342, in __exit__
[rank1]: self.params[0].partition(param_list=self.params, has_been_updated=False)
[rank1]: File "/usr/local/lib/python3.11/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 1487, in partition
[rank1]: self._partition(param_list, has_been_updated=has_been_updated, free_data=True)
[rank1]: File "/usr/local/lib/python3.11/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 1636, in _partition
[rank1]: self._partition_param(param, has_been_updated=has_been_updated, free_data=True)
[rank1]: File "/usr/local/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn
[rank1]: ret_val = func(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/usr/local/lib/python3.11/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 1645, in _partition_param
[rank1]: assert param.ds_status is not ZeroParamStatus.INFLIGHT, f" {param} Cannot partition a param in flight"
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: AssertionError: Parameter containing:
[rank1]: tensor([[-0.0013, -0.0126, -0.0043, ..., 0.0074, 0.0014, -0.0085],
[rank1]: [-0.0018, -0.0095, 0.0056, ..., -0.0013, 0.0281, -0.0211],
[rank1]: [-0.0204, -0.0004, 0.0295, ..., -0.0286, 0.0070, 0.0427],
[rank1]: ...,
Your hardware and system info
Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息，如CUDA版本，系统，GPU型号和torch版本等)
- OS: Ubuntu 22.04
- GPU: 8 × NVIDIA H100 80GB
- NVIDIA Driver: 575.51.03
- CUDA Toolkit: V12.4.131
- swift: 3.11.0.dev0
- vllm: 0.11.0
- torch 2.8.0
在docker容器内运行:
modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope ubuntu22.04-cuda12.8.1-py311-torch2.8.0-vllm0.11.0-modelscope1.31.0-swift3.10.1
Additional context
Add any other context about the problem here(在这里补充其他信息)
一般跑到eval steps 的时候，并且进度条显示val 完成的时候出现报错
补充启动脚本:
export student_model='/DATA/disk1/checkpoints/Qwen3-4B-Base'
export teacher_model='/DATA/disk1/checkpoints/Qwen3-4B-Instruct-2507'
NPROC_PER_NODE=8 \
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
swift rlhf \
--rlhf_type gkd \
--model $student_model \
--teacher_model $teacher_model \
--train_type full \
--dataset '/DATA/disk1/datasets/Math/swift_math_1w_1125.jsonl' \
--split_dataset_ratio 0.01 \
--seq_kd false \
--lmbda 1.0 \
--temperature 1.4 \
--beta 1.0 \
--torch_dtype bfloat16 \
--num_train_epochs 1 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--learning_rate 1e-5 \
--gradient_accumulation_steps 1 \
--eval_steps 1500 \
--save_steps 50 \
--save_total_limit 2 \
--logging_steps 5 \
--max_length 13000 \
--max_completion_length 8192 \
--output_dir output/reverse_kl \
--warmup_ratio 0.05 \
--save_only_model true \
--dataloader_num_workers 4 \
--dataset_num_proc 4 \
--deepspeed zero3 \
--attn_impl flash_attn \
--teacher_deepspeed zero3_offload \
--use_vllm true \
--vllm_mode colocate \
--vllm_gpu_memory_utilization 0.3 \
--sleep_level 1 \
--report_to wandb
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Use gkd/vllm_colocate.sh as the reproduction entry point and run it through the eval step. Start with the traceback path through _move_full_model_to_vllm, _load_state_dict_to_vllm, and vLLM's load_weights, then verify that eval completes without the reported RuntimeError and subsequent DeepSpeed assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100