InternVL3_5-30B-A3B部署失败

我现在在一个5*4090D的服务器上用lmdeploy框架部署InternVL3_5-30B-A3B大模型，依次执行指令如下：
1. docker run -itd --name imdeploy --runtime nvidia --gpus '"device=1,2,3,4"' -e NCCL_P2P_DISABLE=1 -e NCCL_IB_DISABLE=1 -v /llm/models:/models -p 23333:23333 --ipc=host openmmlab/lmdeploy:latest-cu12
2. docker exec -it imdeploy bash
3. lmdeploy serve api_server /models/InternVL3_5-30B-A3B --backend pytorch --tp 4

结果在执行第三条指令时报错，输出信息如下：
2025-09-01 09:20:38,450 ERROR worker.py:429 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::RayWorkerWrapper.warmup_dist() (pid=614, ip=10.11.0.5, actor_id=6543c39add6fefe972f1dad601000000, repr=) File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/opt/py3/lib/python3.10/site-packages/lmdeploy/pytorch/engine/executor/ray_executor.py", line 206, in warmup_dist all_reduce(tmp) File "/opt/py3/lib/python3.10/site-packages/lmdeploy/pytorch/distributed.py", line 248, in all_reduce return dist.all_reduce(tensor, op, group, async_op) File "/opt/py3/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper return func(*args, **kwargs) File "/opt/py3/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2810, in all_reduce work = group.allreduce([tensor], opts) torch.distributed.DistBackendError: NCCL error in: /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:3356, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.26.2 ncclUnhandledCudaError: Call to CUDA function failed. Last error: Cuda failure 217 'peer access is not supported between these two devices'

GPT说是多张卡之间不支持 P2P 通信导致的，但是我在容器内部设置了相关变量，再次运行时还是会报错：
export NCCL_P2P_DISABLE=1
export NCCL_IGNORE_DISABLED_P2P=1
export NCCL_IB_DISABLE=1
export NCCL_NET=Socket
export NCCL_SHM_DISABLE=0
export NCCL_P2P_LEVEL=SYS
export NCCL_NVLS_ENABLE=0

无语了我都，有没有人遇到相同的问题啊