qlora+zero3报错

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

nproc_per_node=2

CUDA_VISIBLE_DEVICES=0,1
NPROC_PER_NODE=$nproc_per_node
swift sft
--model Qwen/Qwen2.5-7B-Instruct
--tuner_type lora
--dataset 'admin/kcentergreedy_verilog_generation_combined_1W#train.jsonl'
--torch_dtype bfloat16
--bnb_4bit_compute_dtype bfloat16
--bnb_4bit_quant_type nf4
--bnb_4bit_use_double_quant true
--quant_method bnb
--quant_bits 4
--num_train_epochs 1
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--learning_rate 1e-4
--lora_rank 8
--lora_alpha 32
--target_modules all-linear
--gradient_accumulation_steps 8
--eval_steps 100
--save_steps 100
--save_total_limit 2
--logging_steps 5
--max_length 12000
--output_dir output
--system 'You are a helpful assistant.'
--warmup_ratio 0.05
--dataloader_num_workers 4
--model_author swift
--model_name swift-robot
--deepspeed zero3 \

How to Reproduce / 如何复现

modelscope-registry.cn-hangzhou.cr.aliyuncs.com/modelscope-repo/modelscope:ubuntu22.04-cuda12.8.1-py311-torch2.8.0-vllm0.11.0-modelscope1.31.0-swift3.10.3
returncode=1, stdout_tail=[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | [rank0]: File "/usr/local/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 118, in log_wrapper | [rank0]: return func(*args, **kwargs) | [rank0]: ^^^^^^^^^^^^^^^^^^^^^ | [rank0]: File "/usr/local/lib/python3.11/site-packages/deepspeed/comm/comm.py", line 318, in all_gather_into_tensor | [rank0]: return cdb.all_gather_into_tensor(output_tensor=output_tensor, input_tensor=tensor, group=group, async_op=async_op) | [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | [rank0]: File "/usr/local/lib/python3.11/site-packages/deepspeed/comm/torch.py", line 233, in all_gather_into_tensor | [rank0]: return self.all_gather_function(output_tensor=output_tensor, | [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | [rank0]: File "/usr/local/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper | [rank0]: return func(*args, **kwargs) | [rank0]: ^^^^^^^^^^^^^^^^^^^^^ | [rank0]: File "/usr/local/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3986, in all_gather_into_tensor | [rank0]: work = group._allgather_base(output_tensor, input_tensor, opts) | [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | [rank0]: TypeError: output tensor must have the same type as input tensor | | Train: 100%|██████████| 93/93 [12:01<00:00, 7.76s/it] | [2026-03-10 16:05:59,144] [ERROR] [launch.py:341:sigkill_handler] ['/usr/local/bin/python', '-u', '/app/finetune-engine/api/train_runner.py', '--local_rank=1', '--json', 'output/Qwen/Qwen2.5-3B-Instruct_sft_lora_3_train_2026310155258/train_args_payload.json'] exits with return code = 1, log=output/Qwen/Qwen2.5-3B-Instruct_sft_lora_3_train_2026310155258/train_stdout.log"

Additional Information / 补充信息

No response

Start by reproducing the reported swift sft command with --deepspeed zero3 in the specified container, then inspect the all_gather_into_tensor traceback in deepspeed/comm/comm.py and torch.distributed.distributed_c10d.py. Done means the QLoRA training run completes without the output/input tensor type mismatch.