swift进行Dora强化训练OOM

Checklist / 检查清单

• I have searched existing issues, and this is a new question or discussion topic. / 我已经搜索过现有的 issues，确认这是一个新的问题与讨论。

Question Description / 问题描述

用swift进行Dora的强化训练出现OOM，请问是模型过大的原因吗，使用的qwen3-30B。
swift版本：3.11.0.dev0
运行gpu：NVIDIA-H800-共8张GPU
运行脚本：
MODEL_PATH="/models/huggingface/Qwen3-30B-Instruct-2507"
MODEL_TYPE="qwen3_moe"

LOG_DIR=/home/RLtrain_swift/swift_scripts/qwen-30B/dora_logs
mkdir -p "$LOG_DIR"
TIME=$(date '+%Y-%m-%d-%H-%M-%S')
LOG_FILE=$LOG_DIR/$(basename "$MODEL_PATH")-${TIME}.log
exec >"$LOG_FILE" 2>&1 # 实时刷盘
echo "[$(date '+%F %T')] 脚本 $0 启动"
echo "[$(date '+%F %T')] 日志：$LOG_FILE"

echo $ZAIP_NODE_NUM
echo $ZAIP_GPUS_PER_NODE
echo $ZAIP_NODE_INDEX
echo $ZAIP_NODE_0_HOST
MASTER_PORT=8006
unset OMPI_COMM_WORLD_LOCAL_RANK
echo "LOCAL_RANK: $LOCAL_RANK"
echo "OMPI_COMM_WORLD_LOCAL_RANK: $OMPI_COMM_WORLD_LOCAL_RANK"

export WANDB_MODE=disabled
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
NNODES=$ZAIP_NODE_NUM
NODE_RANK=$ZAIP_NODE_INDEX
MASTER_ADDR=$ZAIP_NODE_0_HOST
MASTER_PORT=8006
NPROC_PER_NODE=8
swift rlhf
--rlhf_type grpo
--model "$MODEL_PATH"
--model_type "$MODEL_TYPE"
--template qwen3_nothinking
--dataset /dataset/code_14057_gspo-3559_swift.json
--output_dir /home/RLtrain_swift/output/dora_exper/test_dora
--external_plugins /home/swift_learn/grpo_plugin/plugin.py
--reward_funcs external_code1031_reward soft_overlong
--reward_weights 0.8 0.2
--soft_cache_length 1024
--importance_sampling_level sequence
--beta 0.001
--epsilon 3e-4
--epsilon_high 4e-4
--steps_per_generation 4
--use_vllm true
--vllm_mode colocate
--vllm_gpu_memory_utilization 0.4
--vllm_tensor_parallel_size 2
--vllm_max_model_len 4096
--train_type lora
--use_dora true
--torch_dtype bfloat16
--max_length 2048
--max_completion_length 2048
--overlong_filter true
--num_train_epochs 2
--per_device_train_batch_size 1
--learning_rate 1e-5
--gradient_accumulation_steps 4
--save_strategy 'steps'
--save_steps 100
--save_total_limit 10
--logging_steps 1
--warmup_ratio 0.01
--dataloader_num_workers 4
--num_generations 4
--temperature 1.0
--deepspeed zero3
--log_completions true
--sleep_level 1
--offload_model true
--offload_optimizer true
--report_to tensorboard
--num_iterations 1
--move_model_batches 10
报错信息：
Train: 0%| | 1/888 [13:18<196:38:30, 798.10s/it]

{'loss': -0.05080484, 'grad_norm': 0.07277552, 'learning_rate': 1.11e-06, 'completions/mean_length': 1546.875, 'completions/min_length': 445.0, 'completions/max_length': 2048.0, 'completions/clipped_ratio': 0.40625, 'reward': -0.15419665, 'reward_std': 0.53695661, 'frac_reward_zero_std': 0.25, 'rewards/Code1031Reward/mean': -0.04748978, 'rewards/Code1031Reward/std': 1.09906185, 'rewards/SoftOverlong/mean': -0.58102417, 'rewards/SoftOverlong/std': 0.45044452, 'kl': 0.0, 'clip_ratio/low_mean': 0.0, 'clip_ratio/low_min': 0.0, 'clip_ratio/high_mean': 0.0, 'clip_ratio/high_max': 0.0, 'clip_ratio/region_mean': 0.0, 'epoch': 0.0, 'global_step/max_steps': '1/888', 'percentage': '0.11%', 'elapsed_time': '13m 18s', 'remaining_time': '8d 4h 41m 48s', 'memory(GiB)': 69.89, 'train_speed(iter/s)': 0.001253}

Train: 0%| | 1/888 [13:18<196:38:30, 798.10s/it]
Train: 0%| | 1/888 [13:18<196:38:30, 798.10s/it]INFO 01-19 21:02:18 [executor_base.py:205] It took 1.241156 seconds to wake up tags ['weights'].
INFO 01-19 21:02:18 [executor_base.py:205] It took 1.272414 seconds to wake up tags ['weights'].
INFO 01-19 21:02:18 [executor_base.py:205] It took 1.274521 seconds to wake up tags ['weights'].
INFO 01-19 21:02:18 [executor_base.py:205] It took 1.291996 seconds to wake up tags ['weights'].
INFO 01-19 21:02:18 [executor_base.py:205] It took 1.292224 seconds to wake up tags ['weights'].
INFO 01-19 21:02:18 [executor_base.py:205] It took 1.293697 seconds to wake up tags ['weights'].
INFO 01-19 21:02:18 [executor_base.py:205] It took 1.293990 seconds to wake up tags ['weights'].
INFO 01-19 21:02:18 [executor_base.py:205] It took 1.306216 seconds to wake up tags ['weights'].
[rank7]: Traceback (most recent call last):
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/cli/rlhf.py", line 7, in
[rank7]: rlhf_main()
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank7]: return SwiftRLHF(args).main()
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/llm/base.py", line 49, in main
[rank7]: result = self.run()
[rank7]: ^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/ray/base.py", line 170, in wrapper
[rank7]: return func(self, *args, **kwargs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/sft.py", line 207, in run
[rank7]: return self.train(trainer)
[rank7]: ^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/sft.py", line 255, in train
[rank7]: trainer.train(trainer.args.resume_from_checkpoint)
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/mixin.py", line 815, in train
[rank7]: res = super().train(*args, **kwargs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2325, in train
[rank7]: return inner_training_loop(
[rank7]: ^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank7]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1510, in training_step
[rank7]: return super().training_step(model, inputs, num_items_in_batch)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 4014, in training_step
[rank7]: inputs = self._prepare_inputs(inputs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank7]: return func(self, *args, **kwargs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 177, in _prepare_inputs
[rank7]: generation_batch = self._generate_and_score_completions(generation_batch)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank7]: return func(self, *args, **kwargs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 220, in _generate_and_score_completions
[rank7]: inputs = self._generate_completions(inputs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 201, in _generate_completions
[rank7]: results = self._fast_infer(inputs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 640, in _fast_infer
[rank7]: self._move_model_to_vllm()
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank7]: return func(self, *args, **kwargs)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 354, in _move_model_to_vllm
[rank7]: self._move_full_model_to_vllm()
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 466, in _move_full_model_to_vllm
[rank7]: self.model.unmerge_adapter()
[rank7]: File "/usr/local/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 833, in unmerge_adapter
[rank7]: module.unmerge()
[rank7]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 371, in unmerge_patched
[rank7]: return self.unmerge_origin()
[rank7]: ^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/peft/tuners/lora/layer.py", line 704, in unmerge
[rank7]: unmerged = self.lora_variant[active_adapter].unmerge(self, active_adapter, weight)
[rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank7]: File "/usr/local/lib/python3.11/site-packages/peft/tuners/lora/variants.py", line 105, in unmerge
[rank7]: new_weight = orig_weight.data / dora_factor.view(-1, 1) - delta_weight
[rank7]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~
[rank7]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 7 has a total capacity of 79.10 GiB of which 8.00 MiB is free. Process 566684 has 79.00 GiB memory in use. Of the allocated memory 77.41 GiB is allocated by PyTorch, with 184.91 MiB allocated in private pools (e.g., CUDA Graphs), and 43.59 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[rank6]: Traceback (most recent call last):
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/cli/rlhf.py", line 7, in
[rank6]: rlhf_main()
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank6]: return SwiftRLHF(args).main()
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/llm/base.py", line 49, in main
[rank6]: result = self.run()
[rank6]: ^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/ray/base.py", line 170, in wrapper
[rank6]: return func(self, *args, **kwargs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/sft.py", line 207, in run
[rank6]: return self.train(trainer)
[rank6]: ^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/sft.py", line 255, in train
[rank6]: trainer.train(trainer.args.resume_from_checkpoint)
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/mixin.py", line 815, in train
[rank6]: res = super().train(*args, **kwargs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2325, in train
[rank6]: return inner_training_loop(
[rank6]: ^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank6]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1510, in training_step
[rank6]: return super().training_step(model, inputs, num_items_in_batch)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 4014, in training_step
[rank6]: inputs = self._prepare_inputs(inputs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank6]: return func(self, *args, **kwargs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 177, in _prepare_inputs
[rank6]: generation_batch = self._generate_and_score_completions(generation_batch)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank6]: return func(self, *args, **kwargs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 220, in _generate_and_score_completions
[rank6]: inputs = self._generate_completions(inputs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 201, in _generate_completions
[rank6]: results = self._fast_infer(inputs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 640, in _fast_infer
[rank6]: self._move_model_to_vllm()
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank6]: return func(self, *args, **kwargs)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 354, in _move_model_to_vllm
[rank6]: self._move_full_model_to_vllm()
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 466, in _move_full_model_to_vllm
[rank6]: self.model.unmerge_adapter()
[rank6]: File "/usr/local/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 833, in unmerge_adapter
[rank6]: module.unmerge()
[rank6]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 371, in unmerge_patched
[rank6]: return self.unmerge_origin()
[rank6]: ^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/peft/tuners/lora/layer.py", line 704, in unmerge
[rank6]: unmerged = self.lora_variant[active_adapter].unmerge(self, active_adapter, weight)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/usr/local/lib/python3.11/site-packages/peft/tuners/lora/variants.py", line 105, in unmerge
[rank6]: new_weight = orig_weight.data / dora_factor.view(-1, 1) - delta_weight
[rank6]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~
[rank6]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 6 has a total capacity of 79.10 GiB of which 6.00 MiB is free. Process 566683 has 79.00 GiB memory in use. Of the allocated memory 77.41 GiB is allocated by PyTorch, with 184.91 MiB allocated in private pools (e.g., CUDA Graphs), and 46.10 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[rank3]: Traceback (most recent call last):
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/cli/rlhf.py", line 7, in
[rank3]: rlhf_main()
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank3]: return SwiftRLHF(args).main()
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/llm/base.py", line 49, in main
[rank3]: result = self.run()
[rank3]: ^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/ray/base.py", line 170, in wrapper
[rank3]: return func(self, *args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/sft.py", line 207, in run
[rank3]: return self.train(trainer)
[rank3]: ^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/llm/train/sft.py", line 255, in train
[rank3]: trainer.train(trainer.args.resume_from_checkpoint)
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/mixin.py", line 815, in train
[rank3]: res = super().train(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2325, in train
[rank3]: return inner_training_loop(
[rank3]: ^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank3]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1510, in training_step
[rank3]: return super().training_step(model, inputs, num_items_in_batch)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 4014, in training_step
[rank3]: inputs = self._prepare_inputs(inputs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank3]: return func(self, *args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 177, in _prepare_inputs
[rank3]: generation_batch = self._generate_and_score_completions(generation_batch)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank3]: return func(self, *args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 220, in _generate_and_score_completions
[rank3]: inputs = self._generate_completions(inputs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/grpo_trainer.py", line 201, in _generate_completions
[rank3]: results = self._fast_infer(inputs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 640, in _fast_infer
[rank3]: self._move_model_to_vllm()
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 408, in wrapper
[rank3]: return func(self, *args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 354, in _move_model_to_vllm
[rank3]: self._move_full_model_to_vllm()
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/rollout_mixin.py", line 466, in _move_full_model_to_vllm
[rank3]: self.model.unmerge_adapter()
[rank3]: File "/usr/local/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 833, in unmerge_adapter
[rank3]: module.unmerge()
[rank3]: File "/usr/local/lib/python3.11/site-packages/swift/trainers/rlhf_trainer/utils.py", line 371, in unmerge_patched
[rank3]: return self.unmerge_origin()
[rank3]: ^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/peft/tuners/lora/layer.py", line 704, in unmerge
[rank3]: unmerged = self.lora_variant[active_adapter].unmerge(self, active_adapter, weight)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/usr/local/lib/python3.11/site-packages/peft/tuners/lora/variants.py", line 105, in unmerge
[rank3]: new_weight = orig_weight.data / dora_factor.view(-1, 1) - delta_weight
[rank3]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~
[rank3]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 3 has a total capacity of 79.10 GiB of which 6.00 MiB is free. Process 566680 has 79.00 GiB memory in use. Of the allocated memory 77.41 GiB is allocated by PyTorch, with 184.91 MiB allocated in private pools (e.g., CUDA Graphs), and 47.15 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Start with swift/llm/train/rlhf.py and follow the traceback into swift/trainers/rlhf_trainer/rollout_mixin.py, especially _move_full_model_to_vllm, then inspect the PEFT unmerge path shown in the report. Reproduce the supplied swift rlhf command with the Qwen3-30B configuration and determine why unmerge needs another 20 MiB; done means the OOM cause and a verified resolution or limitation are documented.