多卡推理qwen2.5-vl-7B的时候张量并行报错

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 torchrun --nproc-per-node=8 run.py --data MMBench_TEST_EN MMBench_TEST_CN MMStar MME MMMU_TEST Q-Bench1_TEST --model Qwen2.5-VL-7B-Instruct --verbose

报错：
File "/home/zhoupc/anaconda3/envs/qwen/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/zhoupc/anaconda3/envs/qwen/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1845, in _call_impl return inner() ^^^^^^^ File "/home/zhoupc/anaconda3/envs/qwen/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1793, in inner result = forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/zhoupc/anaconda3/envs/qwen/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 725, in forward return self._conv_forward(input, self.weight, self.bias) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/zhoupc/anaconda3/envs/qwen/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 720, in _conv_forward return F.conv3d( ^^^^^^^^^ File "/home/zhoupc/anaconda3/envs/qwen/lib/python3.11/site-packages/torch/_compile.py", line 32, in inner return disable_fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/zhoupc/anaconda3/envs/qwen/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/zhoupc/anaconda3/envs/qwen/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 346, in __torch_dispatch__ return DTensor._op_dispatcher.dispatch( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/zhoupc/anaconda3/envs/qwen/lib/python3.11/site-packages/torch/distributed/tensor/_dispatch.py", line 164, in dispatch return self._custom_op_handlers[op_call](op_call, args, kwargs) # type: ignore[operator] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/zhoupc/anaconda3/envs/qwen/lib/python3.11/site-packages/torch/distributed/tensor/_tp_conv.py", line 238, in convolution_handler dtensor.DTensor._op_dispatcher.sharding_propagator.propagate(op_info) File "/home/zhoupc/anaconda3/envs/qwen/lib/python3.11/site-packages/torch/distributed/tensor/_sharding_prop.py", line 206, in propagate OutputSharding, self.propagate_op_sharding(op_info.schema) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/zhoupc/anaconda3/envs/qwen/lib/python3.11/site-packages/torch/distributed/tensor/_sharding_prop.py", line 46, in __call__ return self.cache(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/zhoupc/anaconda3/envs/qwen/lib/python3.11/site-packages/torch/distributed/tensor/_sharding_prop.py", line 422, in propagate_op_sharding_non_cached raise RuntimeError( RuntimeError: Sharding propagation failed on op Op(op=aten.convolution.default, args_schema=Spec(R on (5208, 3, 2, 14, 14)), Spec(R on (1280, 3, 2, 14, 14)), None, [2, 14, 14], [0, 0, 0], [1, 1, 1], False, [0, 0, 0], 1 @ mesh: (8,)). Error:

可以确认的是显卡是正常的，因为跑llava-1.5-7B完全没问题

Start by reproducing the command through run.py with the Qwen2.5-VL-7B-Instruct model and the eight-GPU torchrun setup. Compare the failing tensor-parallel path with the reported working llava-1.5-7B run; done means the listed evaluation benchmarks complete without the convolution sharding-propagation error.