modelscope / modelscope/ms-swift
TypeError: must be real number, not NoneType
Open
@hjh0119 is already working on this.
Since Dec 3, 2025.
bug
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
GRPO训练的时候会突然中断报错，比如进行到120多步还是好了，过一会就报错了，最新版本的swift，main分支
### model
model: ${oc.env:WORK_DIR}/models/Qwen3-8B
### method
stage: rlhf
rlhf_type: grpo
train_type: full
tuner_backend: peft
deepspeed: zero2
attn_impl: flash_attn
use_liger_kernel: true
### dataset
custom_register_path: randy/register.py
dataset: topo_sel_rlhf#4000
packing: false
max_length: 2048
dataset_num_proc: 16
# split_dataset_ratio: 0.1
### output
output_dir: ${oc.env:WORK_DIR}/train/sft/full/Qwen3-8B
logging_steps: 10
save_strategy: steps
save_steps: 100
save_total_limit: 10
report_to: tensorboard
### train
per_device_train_batch_size: 8
gradient_accumulation_steps: 1
# warmup_ratio: 0.1
learning_rate: 1e-6
num_train_epochs: 1.0
lr_scheduler_type: cosine
reward_funcs: accuracy,format
use_vllm: true
vllm_mode: server
vllm_server_host: 10.239.2.28
vllm_server_port: 8000
max_completion_length : 1024
temperature : 1.0
num_generations: 8
log_completions: true
per_device_eval_batch_size: 8
Train: 35%|████████████████████████████████████▎ | 176/500 [51:41<1:29:50, 16.64s/it]/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/torch/utils/checkpoint.py:85: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn(
/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/torch/utils/checkpoint.py:85: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn(
/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/torch/utils/checkpoint.py:85: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn(
[rank6]: Traceback (most recent call last):
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/cli/rlhf.py", line 7, in <module>
[rank6]: rlhf_main()
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank6]: return SwiftRLHF(args).main()
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/base.py", line 49, in main
[rank6]: result = self.run()
[rank6]: ^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/ray/base.py", line 170, in wrapper
[rank6]: return func(self, *args, **kwargs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 196, in run
[rank6]: return self.train(trainer)
[rank6]: ^^^^^^^^^^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 244, in train
[rank6]: trainer.train(trainer.args.resume_from_checkpoint)
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/mixin.py", line 816, in train
[rank6]: res = super().train(*args, **kwargs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2325, in train
[rank6]: return inner_training_loop(
[rank6]: ^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank6]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1696, in training_step
[rank6]: return super().training_step(model, inputs, num_items_in_batch)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 4014, in training_step
[rank6]: inputs = self._prepare_inputs(inputs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank6]: return func(self, *args, **kwargs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 177, in _prepare_inputs
[rank6]: generation_batch = self._generate_and_score_completions(generation_batch)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank6]: return func(self, *args, **kwargs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 228, in _generate_and_score_completions
[rank6]: batch_encoded_inputs = self._prepare_batch_inputs(inputs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank6]: return func(self, *args, **kwargs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 908, in _prepare_batch_inputs
[rank6]: rollout_logps_tensor[i, completion_indices] = torch.tensor(
[rank6]: ^^^^^^^^^^^^^
[rank6]: TypeError: must be real number, not NoneType
[rank2]: Traceback (most recent call last):
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/cli/rlhf.py", line 7, in <module>
[rank2]: rlhf_main()
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank2]: return SwiftRLHF(args).main()
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/base.py", line 49, in main
[rank2]: result = self.run()
[rank2]: ^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/ray/base.py", line 170, in wrapper
[rank2]: return func(self, *args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 196, in run
[rank2]: return self.train(trainer)
[rank2]: ^^^^^^^^^^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 244, in train
[rank2]: trainer.train(trainer.args.resume_from_checkpoint)
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/mixin.py", line 816, in train
[rank2]: res = super().train(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2325, in train
[rank2]: return inner_training_loop(
[rank2]: ^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank2]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1696, in training_step
[rank2]: return super().training_step(model, inputs, num_items_in_batch)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 4014, in training_step
[rank2]: inputs = self._prepare_inputs(inputs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank2]: return func(self, *args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 177, in _prepare_inputs
[rank2]: generation_batch = self._generate_and_score_completions(generation_batch)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank2]: return func(self, *args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 228, in _generate_and_score_completions
[rank2]: batch_encoded_inputs = self._prepare_batch_inputs(inputs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank2]: return func(self, *args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 908, in _prepare_batch_inputs
[rank2]: rollout_logps_tensor[i, completion_indices] = torch.tensor(
[rank2]: ^^^^^^^^^^^^^
[rank2]: TypeError: must be real number, not NoneType
[rank4]: Traceback (most recent call last):
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/cli/rlhf.py", line 7, in <module>
[rank4]: rlhf_main()
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank4]: return SwiftRLHF(args).main()
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/base.py", line 49, in main
[rank4]: result = self.run()
[rank4]: ^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/ray/base.py", line 170, in wrapper
[rank4]: return func(self, *args, **kwargs)
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 196, in run
[rank4]: return self.train(trainer)
[rank4]: ^^^^^^^^^^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 244, in train
[rank4]: trainer.train(trainer.args.resume_from_checkpoint)
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/mixin.py", line 816, in train
[rank4]: res = super().train(*args, **kwargs)
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2325, in train
[rank4]: return inner_training_loop(
[rank4]: ^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank4]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1696, in training_step
[rank4]: return super().training_step(model, inputs, num_items_in_batch)
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 4014, in training_step
[rank4]: inputs = self._prepare_inputs(inputs)
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank4]: return func(self, *args, **kwargs)
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 177, in _prepare_inputs
[rank4]: generation_batch = self._generate_and_score_completions(generation_batch)
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank4]: return func(self, *args, **kwargs)
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 228, in _generate_and_score_completions
[rank4]: batch_encoded_inputs = self._prepare_batch_inputs(inputs)
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank4]: return func(self, *args, **kwargs)
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 908, in _prepare_batch_inputs
[rank4]: rollout_logps_tensor[i, completion_indices] = torch.tensor(
[rank4]: ^^^^^^^^^^^^^
[rank4]: TypeError: must be real number, not NoneType
[INFO:swift] last_model_checkpoint: /nas_train/app.e0016372/train/sft/full/Qwen3-8B/v3-20251203-123948/checkpoint-100
[INFO:swift] best_model_checkpoint: None
[INFO:swift] images_dir: /nas_train/app.e0016372/train/sft/full/Qwen3-8B/v3-20251203-123948/images
[rank7]: Traceback (most recent call last):
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/cli/rlhf.py", line 7, in <module>
[rank7]: rlhf_main()
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank7]: return SwiftRLHF(args).main()
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/base.py", line 49, in main
[rank7]: result = self.run()
[rank7]: ^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/ray/base.py", line 170, in wrapper
[rank7]: return func(self, *args, **kwargs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 196, in run
[rank7]: return self.train(trainer)
[rank7]: ^^^^^^^^^^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 244, in train
[rank7]: trainer.train(trainer.args.resume_from_checkpoint)
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/mixin.py", line 816, in train
[rank7]: res = super().train(*args, **kwargs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2325, in train
[rank7]: return inner_training_loop(
[rank7]: ^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank7]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1696, in training_step
[rank7]: return super().training_step(model, inputs, num_items_in_batch)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 4014, in training_step
[rank7]: inputs = self._prepare_inputs(inputs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank7]: return func(self, *args, **kwargs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 177, in _prepare_inputs
[rank7]: generation_batch = self._generate_and_score_completions(generation_batch)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank7]: return func(self, *args, **kwargs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 228, in _generate_and_score_completions
[rank7]: batch_encoded_inputs = self._prepare_batch_inputs(inputs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank7]: return func(self, *args, **kwargs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 908, in _prepare_batch_inputs
[rank7]: rollout_logps_tensor[i, completion_indices] = torch.tensor(
[rank7]: ^^^^^^^^^^^^^
[rank7]: TypeError: must be real number, not NoneType
[rank1]: Traceback (most recent call last):
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/cli/rlhf.py", line 7, in <module>
[rank1]: rlhf_main()
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank1]: return SwiftRLHF(args).main()
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/base.py", line 49, in main
[rank1]: result = self.run()
[rank1]: ^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/ray/base.py", line 170, in wrapper
[rank1]: return func(self, *args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 196, in run
[rank1]: return self.train(trainer)
[rank1]: ^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 244, in train
[rank1]: trainer.train(trainer.args.resume_from_checkpoint)
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/mixin.py", line 816, in train
[rank1]: res = super().train(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2325, in train
[rank1]: return inner_training_loop(
[rank1]: ^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank1]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1696, in training_step
[rank1]: return super().training_step(model, inputs, num_items_in_batch)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 4014, in training_step
[rank1]: inputs = self._prepare_inputs(inputs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank1]: return func(self, *args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 177, in _prepare_inputs
[rank1]: generation_batch = self._generate_and_score_completions(generation_batch)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank1]: return func(self, *args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 228, in _generate_and_score_completions
[rank1]: batch_encoded_inputs = self._prepare_batch_inputs(inputs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank1]: return func(self, *args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 908, in _prepare_batch_inputs
[rank1]: rollout_logps_tensor[i, completion_indices] = torch.tensor(
[rank1]: ^^^^^^^^^^^^^
[rank1]: TypeError: must be real number, not NoneType
[rank5]: Traceback (most recent call last):
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/cli/rlhf.py", line 7, in <module>
[rank5]: rlhf_main()
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank5]: return SwiftRLHF(args).main()
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/base.py", line 49, in main
[rank5]: result = self.run()
[rank5]: ^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/ray/base.py", line 170, in wrapper
[rank5]: return func(self, *args, **kwargs)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 196, in run
[rank5]: return self.train(trainer)
[rank5]: ^^^^^^^^^^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 244, in train
[rank5]: trainer.train(trainer.args.resume_from_checkpoint)
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/mixin.py", line 816, in train
[rank5]: res = super().train(*args, **kwargs)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2325, in train
[rank5]: return inner_training_loop(
[rank5]: ^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank5]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1696, in training_step
[rank5]: return super().training_step(model, inputs, num_items_in_batch)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 4014, in training_step
[rank5]: inputs = self._prepare_inputs(inputs)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank5]: return func(self, *args, **kwargs)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 177, in _prepare_inputs
[rank5]: generation_batch = self._generate_and_score_completions(generation_batch)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank5]: return func(self, *args, **kwargs)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 228, in _generate_and_score_completions
[rank5]: batch_encoded_inputs = self._prepare_batch_inputs(inputs)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank5]: return func(self, *args, **kwargs)
[rank5]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank5]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 908, in _prepare_batch_inputs
[rank5]: rollout_logps_tensor[i, completion_indices] = torch.tensor(
[rank5]: ^^^^^^^^^^^^^
[rank5]: TypeError: must be real number, not NoneType
[rank3]: Traceback (most recent call last):
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/cli/rlhf.py", line 7, in <module>
[rank3]: rlhf_main()
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank3]: return SwiftRLHF(args).main()
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/base.py", line 49, in main
[rank3]: result = self.run()
[rank3]: ^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/ray/base.py", line 170, in wrapper
[rank3]: return func(self, *args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 196, in run
[rank3]: return self.train(trainer)
[rank3]: ^^^^^^^^^^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 244, in train
[rank3]: trainer.train(trainer.args.resume_from_checkpoint)
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/mixin.py", line 816, in train
[rank3]: res = super().train(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2325, in train
[rank3]: return inner_training_loop(
[rank3]: ^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank3]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1696, in training_step
[rank3]: return super().training_step(model, inputs, num_items_in_batch)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 4014, in training_step
[rank3]: inputs = self._prepare_inputs(inputs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank3]: return func(self, *args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 177, in _prepare_inputs
[rank3]: generation_batch = self._generate_and_score_completions(generation_batch)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank3]: return func(self, *args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 228, in _generate_and_score_completions
[rank3]: batch_encoded_inputs = self._prepare_batch_inputs(inputs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank3]: return func(self, *args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 908, in _prepare_batch_inputs
[rank3]: rollout_logps_tensor[i, completion_indices] = torch.tensor(
[rank3]: ^^^^^^^^^^^^^
[rank3]: TypeError: must be real number, not NoneType
Exception ignored in: <function DeepSpeedEngine.__del__ at 0x7fd361ae63e0>
Traceback (most recent call last):
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 519, in __del__
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 523, in destroy
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/bf16_optimizer.py", line 107, in destroy
IndexError: list index out of range
Exception ignored in: <function DeepSpeedEngine.__del__ at 0x7f8e2bd163e0>
Traceback (most recent call last):
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 519, in __del__
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 523, in destroy
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/bf16_optimizer.py", line 107, in destroy
IndexError: list index out of range
Exception ignored in: <function DeepSpeedEngine.__del__ at 0x7fa015bae3e0>
Traceback (most recent call last):
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 519, in __del__
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 523, in destroy
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/bf16_optimizer.py", line 107, in destroy
IndexError: list index out of range
Exception ignored in: <function DeepSpeedEngine.__del__ at 0x7fac5452e3e0>
Traceback (most recent call last):
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 519, in __del__
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 523, in destroy
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/bf16_optimizer.py", line 107, in destroy
IndexError: list index out of range
Exception ignored in: <function DeepSpeedEngine.__del__ at 0x7fbe8c2c63e0>
Traceback (most recent call last):
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 519, in __del__
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 523, in destroy
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/bf16_optimizer.py", line 107, in destroy
IndexError: list index out of range
Exception ignored in: <function DeepSpeedEngine.__del__ at 0x7fa81f4423e0>
Traceback (most recent call last):
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 519, in __del__
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 523, in destroy
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/bf16_optimizer.py", line 107, in destroy
IndexError: list index out of range
Exception ignored in: <function DeepSpeedEngine.__del__ at 0x7f80541163e0>
Traceback (most recent call last):
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 519, in __del__
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 523, in destroy
File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/deepspeed/runtime/bf16_optimizer.py", line 107, in destroy
IndexError: list index out of range
[rank0]: Traceback (most recent call last):
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/cli/rlhf.py", line 7, in <module>
[rank0]: rlhf_main()
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank0]: return SwiftRLHF(args).main()
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/base.py", line 49, in main
[rank0]: result = self.run()
[rank0]: ^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/ray/base.py", line 170, in wrapper
[rank0]: return func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 196, in run
[rank0]: return self.train(trainer)
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/llm/train/sft.py", line 244, in train
[rank0]: trainer.train(trainer.args.resume_from_checkpoint)
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/mixin.py", line 816, in train
[rank0]: res = super().train(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2325, in train
[rank0]: return inner_training_loop(
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank0]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1696, in training_step
[rank0]: return super().training_step(model, inputs, num_items_in_batch)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/miniforge3/envs/swift/lib/python3.12/site-packages/transformers/trainer.py", line 4014, in training_step
[rank0]: inputs = self._prepare_inputs(inputs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank0]: return func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 177, in _prepare_inputs
[rank0]: generation_batch = self._generate_and_score_completions(generation_batch)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank0]: return func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 228, in _generate_and_score_completions
[rank0]: batch_encoded_inputs = self._prepare_batch_inputs(inputs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank0]: return func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/nas_train/app.e0016372/projects/ms-swift/swift/trainers/rlhf_trainer/grpo_trainer.py", line 908, in _prepare_batch_inputs
[rank0]: rollout_logps_tensor[i, completion_indices] = torch.tensor(
[rank0]: ^^^^^^^^^^^^^
[rank0]: TypeError: must be real number, not NoneType
[2025-12-03 13:32:51,088] [ERROR] [launch.py:341:sigkill_handler] ['/nas_train/app.e0016372/miniforge3/envs/swift/bin/python3', '-u', 'swift/cli/rlhf.py', '--local_rank=7', '--model', '/nas_train/app.e0016372/models/Qwen3-8B', '--rlhf_type', 'grpo', '--train_type', 'full', '--tuner_backend', 'peft', '--deepspeed', 'zero2', '--attn_impl', 'flash_attn', '--use_liger_kernel', 'True', '--custom_register_path', 'randy/register.py', '--dataset', 'topo_sel_rlhf#4000', '--packing', 'False', '--max_length', '2048', '--dataset_num_proc', '16', '--output_dir', '/nas_train/app.e0016372/train/sft/full/Qwen3-8B', '--logging_steps', '10', '--save_strategy', 'steps', '--save_steps', '100', '--save_total_limit', '10', '--report_to', 'tensorboard', '--per_device_train_batch_size', '8', '--gradient_accumulation_steps', '1', '--learning_rate', '1e-06', '--num_train_epochs', '1.0', '--lr_scheduler_type', 'cosine', '--reward_funcs', 'accuracy', 'format', '--use_vllm', 'True', '--vllm_mode', 'server', '--vllm_server_host', '10.239.2.28', '--vllm_server_port', '8000', '--max_completion_length', '1024', '--temperature', '1.0', '--num_generations', '8', '--log_completions', 'True', '--per_device_eval_batch_size', '8'] exits with return code = 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.