modelscope / modelscope/ms-swift

GKD训练报错

Open
#6,575 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

stale
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

指令

rollout：
NPROC_PER_NODE=8
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
swift rollout
--model Qwen3-VL-30B-A3B-Instruct
--vllm_max_model_len 24000
--vllm_tensor_parallel_size 4
--vllm_data_parallel_size 2
--vllm_gpu_memory_utilization 0.9
--port 8002
client
NPROC_PER_NODE=8
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
swift rlhf
--rlhf_type gkd
--model Qwen3-VL-30B-A3B-Instruct
--teacher_model Qwen3-VL-30B-A3B-Instruct
--train_type full
--dataset data_10w.jsonl
--seq_kd false
--lmbda 1
--beta 1
--torch_dtype bfloat16
--use_vllm true
--vllm_mode server
--vllm_server_host ** **
--vllm_server_port 8002 8002
--num_train_epochs 1
--per_device_train_batch_size 1
--learning_rate 1e-6
--gradient_accumulation_steps 2
--save_steps 25
--save_total_limit 6
--logging_steps 1
--max_length 24000
--max_completion_length 8192
--output_dir output
--warmup_ratio 0.05
--save_only_model true
--dataloader_num_workers 32
--dataset_num_proc 32
--deepspeed zero3
--teacher_deepspeed zero3
--offload_teacher_model true
--attn_impl flash_attn

报错

rollout
[1;36m(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] Traceback (most recent call last):
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/vllm/v1/executor/multiproc_executor.py", line 666, in worker_busy_loop
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] output = func(*args, **kwargs)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/vllm/v1/worker/gpu_worker.py", line 491, in execute_dummy_batch
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] self.model_runner.dummy_run(1, uniform_decode=True)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/torch/utils/contextlib.py", line 120, in decorate_context
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] return func(*args, **kwargs)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3152, in dummy_run
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] outputs = self.model(
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/vllm/compilation/cuda_graph.py", line 121, in call
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] return self.runnable(*args, **kwargs)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in wrapped_call_impl
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] return self.call_impl(*args, **kwargs)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in call_impl
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] return forward_call(*args, **kwargs)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/qwen3_vl.py", line 1576, in forward
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] hidden_states = self.language_model.model(
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/vllm/compilation/decorators.py", line 317, in call
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] model_output = self.forward(*args, **kwargs)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/vllm/model_executor/models/qwen3_vl_moe.py", line 82, in forward
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] def forward(
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/torch/dynamo/eval_frame.py", line 375, in call
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] return super().call(*args, **kwargs)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in wrapped_call_impl
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] return self.call_impl(*args, **kwargs)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in call_impl
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] return forward_call(*args, **kwargs)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/torch/dynamo/eval_frame.py", line 929, in fn
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] return fn(*args, **kwargs)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/torch/fx/graph_module.py", line 848, in call_wrapped
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] return self.wrapped_call(self, *args, **kwargs)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/torch/fx/graph_module.py", line 424, in call
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] raise e
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/torch/fx/graph_module.py", line 411, in call
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc]
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in wrapped_call_impl
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] return self.call_impl(*args, **kwargs)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in call_impl
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] return forward_call(*args, **kwargs)
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] File "<eval_with_key>.98", line 357, in forward
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] submod_2 = self.submod_2(getitem_3, s59, l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_parameters_weight, l_inputs_embeds, l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight, l_self_modules_layers_modules_0_modules_mlp_modules_gate_parameters_weight, l_deepstack_input_embeds_tensors_deepstack_input_embeds_0, l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight, l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_weight, s18, l_self_modules_layers_modules_1_modules_self_attn_modules_q_norm_parameters_weight, l_self_modules_layers_modules_1_modules_self_attn_modules_k_norm_parameters_weight, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache, l_positions, s7, getitem_4, getitem_5, getitem_6); getitem_3 = l_self_modules_layers_modules_0_modules_self_attn_modules_o_proj_parameters_weight = l_inputs_embeds = l_self_modules_layers_modules_0_modules_post_attention_layernorm_parameters_weight = l_self_modules_layers_modules_0_modules_mlp_modules_gate_parameters_weight = l_deepstack_input_embeds_tensors_deepstack_input_embeds_0 = l_self_modules_layers_modules_1_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_qkv_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_1_modules_self_attn_modules_k_norm_parameters_weight_ = None
(Worker_DP1_TP0 pid=1170) ERROR 11-13 08:09:34 [multiproc_executor.py:671] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

client：
/usr/local/lib/python3.11/site-packages/swift/utils/tb_utils.py:73: UserWarning: Glyph 8722 (\N{MINUS SIGN}) missing from font(s) SimHei.
plt.savefig(fpath, dpi=dpi, bbox_inches='tight')
[rank0]: Traceback (most recent call last):
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/cli/rlhf.py", line 7, in
[rank0]: rlhf_main()
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/rlhf.py", line 227, in rlhf_main
[rank0]: return SwiftRLHF(args).main()
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/llm/base.py", line 49, in main
[rank0]: result = self.run()
[rank0]: ^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/ray/base.py", line 170, in wrapper
[rank0]: return func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/sft.py", line 206, in run
[rank0]: return self.train(trainer)
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/sft.py", line 254, in train
[rank0]: trainer.train(trainer.args.resume_from_checkpoint)
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/mixin.py", line 815, in train
[rank0]: res = super().train(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2325, in train
[rank0]: return inner_training_loop(
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank0]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 407, in wrapper
[rank0]: return func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/gkd_trainer.py", line 202, in training_step
[rank0]: generated_inputs = self._fast_infer(processed_inputs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 671, in _fast_infer
[rank0]: outputs = self._infer_single_or_multi_turn(inputs, self.request_config)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 519, in _infer_single_or_multi_turn
[rank0]: rollout_outputs: List[RolloutOutput] = self._rollout(inputs, request_config, is_global_inputs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 481, in _rollout
[rank0]: rollout_outputs = self._server_rollout(inputs, request_config, is_global_inputs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 726, in _server_rollout
[rank0]: all_outputs: List[RolloutOutput] = self._engine_infer(
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 793, in _engine_infer
[rank0]: res = self.vllm_client.infer([asdict(req) for req in infer_requests],
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/vllm_client.py", line 174, in infer
[rank0]: raise RuntimeError(f'Multiple errors: {all_errors}')
[rank0]: RuntimeError: Multiple errors: [Exception('Server 0 failed: 500, Internal Server Error')]

Train: 50%|████▉ | 194/391 [6:38:22<6:44:31, 123.21s/it]
W1113 08:14:39.408000 103 usr/local/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 227 closing signal SIGTERM
W1113 08:14:39.409000 103 usr/local/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 228 closing signal SIGTERM
W1113 08:14:39.410000 103 usr/local/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 229 closing signal SIGTERM
W1113 08:14:39.413000 103 usr/local/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 230 closing signal SIGTERM
W1113 08:14:39.415000 103 usr/local/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 231 closing signal SIGTERM
W1113 08:14:39.417000 103 usr/local/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 232 closing signal SIGTERM
W1113 08:14:39.421000 103 usr/local/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 233 closing signal SIGTERM
E1113 08:14:39.457000 103 usr/local/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: 1) local_rank: 0 (pid: 226) of binary: /usr/local/bin/python
Traceback (most recent call last):

版本：
pip show ms-swift
Name: ms_swift
Version: 3.11.0.dev0
Summary: Swift: Scalable lightWeight Infrastructure for Fine-Tuning
Home-page: https://github.com/modelscope/swift

pip show vllm
Name: vllm
Version: 0.11.0
Summary: A high-throughput and memory-efficient inference and serving engine for LLMs
Home-page: https://github.com/vllm-project/vllm

看着像是报错在VLLM中

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the supplied swift rollout and swift rlhf commands, then trace the failure from vllm/v1/worker/gpu_model_runner.py through qwen3_vl.py and qwen3_vl_moe.py. Compare the Qwen3-VL GKD configuration with the reported worker traceback; done means the rollout and GKD client start without this error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.