kvcache offload failed for rayclusterfleet

### 🐛 Describe the bug

hi，
dose kvcache offload support rayclusterfleet？

I try l2 kvcache from https://aibrix.readthedocs.io/latest/features/kvcache-offloading.html#l2-cache-example，but the raycluster head pod raised error bellow：
```
ERROR 08-22 01:16:55 [core.py:632] EngineCore failed to start.
ERROR 08-22 01:16:55 [core.py:632] Traceback (most recent call last):
ERROR 08-22 01:16:55 [core.py:632] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 623, in run_engine_core
ERROR 08-22 01:16:55 [core.py:632] engine_core = EngineCoreProc(*args, **kwargs)
ERROR 08-22 01:16:55 [core.py:632] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-22 01:16:55 [core.py:632] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 441, in __init__
ERROR 08-22 01:16:55 [core.py:632] super().__init__(vllm_config, executor_class, log_stats,
ERROR 08-22 01:16:55 [core.py:632] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 77, in __init__
ERROR 08-22 01:16:55 [core.py:632] self.model_executor = executor_class(vllm_config)
ERROR 08-22 01:16:55 [core.py:632] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-22 01:16:55 [core.py:632] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 263, in __init__
ERROR 08-22 01:16:55 [core.py:632] super().__init__(*args, **kwargs)
ERROR 08-22 01:16:55 [core.py:632] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 53, in __init__
ERROR 08-22 01:16:55 [core.py:632] self._init_executor()
ERROR 08-22 01:16:55 [core.py:632] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/ray_distributed_executor.py", line 47, in _init_executor
ERROR 08-22 01:16:55 [core.py:632] super()._init_executor()
ERROR 08-22 01:16:55 [core.py:632] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/ray_distributed_executor.py", line 107, in _init_executor
ERROR 08-22 01:16:55 [core.py:632] self._init_workers_ray(placement_group)
ERROR 08-22 01:16:55 [core.py:632] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/ray_distributed_executor.py", line 377, in _init_workers_ray
ERROR 08-22 01:16:55 [core.py:632] self._run_workers("init_device")
ERROR 08-22 01:16:55 [core.py:632] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/ray_distributed_executor.py", line 503, in _run_workers
ERROR 08-22 01:16:55 [core.py:632] ray_worker_outputs = ray.get(ray_worker_outputs)
ERROR 08-22 01:16:55 [core.py:632] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-22 01:16:55 [core.py:632] File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
ERROR 08-22 01:16:55 [core.py:632] return fn(*args, **kwargs)
ERROR 08-22 01:16:55 [core.py:632] ^^^^^^^^^^^^^^^^^^^
ERROR 08-22 01:16:55 [core.py:632] File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
ERROR 08-22 01:16:55 [core.py:632] return func(*args, **kwargs)
ERROR 08-22 01:16:55 [core.py:632] ^^^^^^^^^^^^^^^^^^^^^
ERROR 08-22 01:16:55 [core.py:632] File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2858, in get
ERROR 08-22 01:16:55 [core.py:632] values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
ERROR 08-22 01:16:55 [core.py:632] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-22 01:16:55 [core.py:632] File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 960, in get_objects
ERROR 08-22 01:16:55 [core.py:632] raise value
ERROR 08-22 01:16:55 [core.py:632] ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
ERROR 08-22 01:16:55 [core.py:632] class_name: RayWorkerWrapper
ERROR 08-22 01:16:55 [core.py:632] actor_id: 6eab11ab396989fa58b09a6901000000
ERROR 08-22 01:16:55 [core.py:632] pid: 262
ERROR 08-22 01:16:55 [core.py:632] namespace: c7bdd54c-6027-4602-807f-8e820cd06796
ERROR 08-22 01:16:55 [core.py:632] ip: 172.29.130.200
ERROR 08-22 01:16:55 [core.py:632] The actor died because its node has died. Node Id: 3e5823179afbefaa420c013624c4ca47763d22d65b858419c0c5d87b
ERROR 08-22 01:16:55 [core.py:632] the actor's node was terminated expectedly: received SIGTERM
Traceback (most recent call last):
File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 636, in run_engine_core
raise e
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 623, in run_engine_core
engine_core = EngineCoreProc(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 441, in __init__
super().__init__(vllm_config, executor_class, log_stats,
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 77, in __init__
self.model_executor = executor_class(vllm_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 263, in __init__
super().__init__(*args, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 53, in __init__
self._init_executor()
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/ray_distributed_executor.py", line 47, in _init_executor
super()._init_executor()
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/ray_distributed_executor.py", line 107, in _init_executor
self._init_workers_ray(placement_group)
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/ray_distributed_executor.py", line 377, in _init_workers_ray
self._run_workers("init_device")
File "/usr/local/lib/python3.12/dist-packages/vllm/executor/ray_distributed_executor.py", line 503, in _run_workers
ray_worker_outputs = ray.get(ray_worker_outputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 2858, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 960, in get_objects
raise value
ray.exceptions.ActorDiedError: The actor died unexpectedly before finishing this task.
class_name: RayWorkerWrapper
actor_id: 6eab11ab396989fa58b09a6901000000
pid: 262
namespace: c7bdd54c-6027-4602-807f-8e820cd06796
ip: 172.29.130.200
The actor died because its node has died. Node Id: 3e5823179afbefaa420c013624c4ca47763d22d65b858419c0c5d87b
the actor's node was terminated expectedly: received SIGTERM
INFO 08-22 01:16:55 [ray_distributed_executor.py:120] Shutting down Ray distributed executor. If you see error log from logging.cc regarding SIGTERM received, please ignore because this is the expected termination process in Ray.
2025-08-22 01:16:56,086 ERROR worker.py:427 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::RayWorkerWrapper.execute_method() (pid=2114, ip=172.29.160.14, actor_id=f6832a12b5ff7394c3f6c39701000000, repr=)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 620, in execute_method
raise e
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 611, in execute_method
return run_method(self, method, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2985, in run_method
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 603, in init_device
self.worker.init_device() # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 181, in init_device
init_worker_distributed_environment(self.vllm_config, self.rank,
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 581, in init_worker_distributed_environment
ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1121, in ensure_model_parallel_initialized
initialize_model_parallel(tensor_model_parallel_size,
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1063, in initialize_model_parallel
_TP = init_model_parallel_group(group_ranks,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 858, in init_model_parallel_group
return GroupCoordinator(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 257, in __init__
self.device_communicator = device_comm_cls(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 59, in __init__
self.ca_comm = CustomAllreduce(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 146, in __init__
if not current_platform.is_rocm() and not _can_p2p(rank, world_size):
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 38, in _can_p2p
if not gpu_p2p_access_check(rank, i):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 245, in gpu_p2p_access_check
get_world_group().barrier()
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 782, in barrier
torch.distributed.barrier(group=self.cpu_group)
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 4640, in barrier
work.wait()
RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [172.29.130.200]:38832
2025-08-22 01:16:56,088 ERROR worker.py:427 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::RayWorkerWrapper.execute_method() (pid=2123, ip=172.29.160.14, actor_id=2cb49abf493945be985caaea01000000, repr=)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 620, in execute_method
raise e
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 611, in execute_method
return run_method(self, method, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2985, in run_method
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 603, in init_device
self.worker.init_device() # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 181, in init_device
init_worker_distributed_environment(self.vllm_config, self.rank,
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 581, in init_worker_distributed_environment
ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1121, in ensure_model_parallel_initialized
initialize_model_parallel(tensor_model_parallel_size,
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1063, in initialize_model_parallel
_TP = init_model_parallel_group(group_ranks,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 858, in init_model_parallel_group
return GroupCoordinator(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 257, in __init__
self.device_communicator = device_comm_cls(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 59, in __init__
self.ca_comm = CustomAllreduce(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 146, in __init__
if not current_platform.is_rocm() and not _can_p2p(rank, world_size):
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 38, in _can_p2p
if not gpu_p2p_access_check(rank, i):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 245, in gpu_p2p_access_check
get_world_group().barrier()
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 782, in barrier
torch.distributed.barrier(group=self.cpu_group)
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 4640, in barrier
work.wait()
RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [172.29.130.200]:38832
2025-08-22 01:16:56,088 ERROR worker.py:427 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::RayWorkerWrapper.execute_method() (pid=2117, ip=172.29.160.14, actor_id=b94d65acf4604aa5e3d246a201000000, repr=)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 620, in execute_method
raise e
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 611, in execute_method
return run_method(self, method, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2985, in run_method
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 603, in init_device
self.worker.init_device() # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 181, in init_device
init_worker_distributed_environment(self.vllm_config, self.rank,
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 581, in init_worker_distributed_environment
ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1121, in ensure_model_parallel_initialized
initialize_model_parallel(tensor_model_parallel_size,
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1063, in initialize_model_parallel
_TP = init_model_parallel_group(group_ranks,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 858, in init_model_parallel_group
return GroupCoordinator(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 257, in __init__
self.device_communicator = device_comm_cls(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 59, in __init__
self.ca_comm = CustomAllreduce(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 146, in __init__
if not current_platform.is_rocm() and not _can_p2p(rank, world_size):
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 38, in _can_p2p
if not gpu_p2p_access_check(rank, i):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 245, in gpu_p2p_access_check
get_world_group().barrier()
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 782, in barrier
torch.distributed.barrier(group=self.cpu_group)
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 4640, in barrier
work.wait()
RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [172.29.130.200]:38832
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] Error executing method 'init_device'. This might cause deadlock in distributed execution.
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] Traceback (most recent call last):
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 611, in execute_method
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] return run_method(self, method, args, kwargs)
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2985, in run_method
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] return func(*args, **kwargs)
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] return method(self, *_args, **_kwargs)
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 603, in init_device
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] self.worker.init_device() # type: ignore
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 181, in init_device
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] init_worker_distributed_environment(self.vllm_config, self.rank,
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 581, in init_worker_distributed_environment
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1121, in ensure_model_parallel_initialized
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] initialize_model_parallel(tensor_model_parallel_size,
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1063, in initialize_model_parallel
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] _TP = init_model_parallel_group(group_ranks,
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 858, in init_model_parallel_group
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] return GroupCoordinator(
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 257, in __init__
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] self.device_communicator = device_comm_cls(
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 59, in __init__
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] self.ca_comm = CustomAllreduce(
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 146, in __init__
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] if not current_platform.is_rocm() and not _can_p2p(rank, world_size):
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 38, in _can_p2p
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] if not gpu_p2p_access_check(rank, i):
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 245, in gpu_p2p_access_check
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] get_world_group().barrier()
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 782, in barrier
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] torch.distributed.barrier(group=self.cpu_group)
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] return func(*args, **kwargs)
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 4640, in barrier
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] work.wait()
(RayWorkerWrapper pid=2117) ERROR 08-22 01:16:56 [worker_base.py:619] RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [172.29.130.200]:38832
(RayWorkerWrapper pid=2126) INFO 08-22 01:16:41 [__init__.py:1375] Found nccl from library libnccl.so.2 [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) INFO 08-22 01:16:41 [pynccl.py:70] vLLM is using nccl==2.26.2 [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO Bootstrap: Using eth0:172.29.160.14<0> [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO cudaDriverVersion 12080 [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO NCCL version 2.26.2+cuda12.2 [repeated 15x across cluster]
2025-08-22 01:16:56,191 ERROR worker.py:427 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::RayWorkerWrapper.execute_method() (pid=2126, ip=172.29.160.14, actor_id=39cbf410371d8dd2c98e7fa801000000, repr=)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 620, in execute_method
raise e
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 611, in execute_method
return run_method(self, method, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2985, in run_method
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 603, in init_device
self.worker.init_device() # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 181, in init_device
init_worker_distributed_environment(self.vllm_config, self.rank,
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 581, in init_worker_distributed_environment
ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1121, in ensure_model_parallel_initialized
initialize_model_parallel(tensor_model_parallel_size,
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1063, in initialize_model_parallel
_TP = init_model_parallel_group(group_ranks,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 858, in init_model_parallel_group
return GroupCoordinator(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 257, in __init__
self.device_communicator = device_comm_cls(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 59, in __init__
self.ca_comm = CustomAllreduce(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 146, in __init__
if not current_platform.is_rocm() and not _can_p2p(rank, world_size):
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 38, in _can_p2p
if not gpu_p2p_access_check(rank, i):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 245, in gpu_p2p_access_check
get_world_group().barrier()
File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 782, in barrier
torch.distributed.barrier(group=self.cpu_group)
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 4640, in barrier
work.wait()
RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [172.29.160.14]:36537
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO NET/Plugin: Could not find: libnccl-net-none.so. Using internal net plugin. [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [1]mlx5_1:1/IB [2]mlx5_2:1/IB [3]mlx5_3:1/IB [4]mlx5_4:1/IB [5]mlx5_5:1/IB [6]mlx5_6:1/IB [7]mlx5_7:1/IB [RO]; OOB eth0:172.29.160.14<0> [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO Using network IB [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO ncclCommInitRank comm 0xf130850 rank 1 nranks 2 cudaDev 5 nvmlDev 5 busId ab000 commId 0x6a406463fe4882ae - Init START [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO RAS client listening socket at ::1<28028> [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO Bootstrap timings total 0.001224 (create 0.000057, send 0.000192, recv 0.000234, ring 0.000040, delay 0.000001) [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO NCCL_CUMEM_ENABLE set by environment to 0. [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO MNNVL busId 0xab000 fabric UUID 0.0 cliqueId 0x0 state 3 healthMask 0x0 [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO Setting affinity for GPU 5 to ffffffff,ffff0000,00000000,ffffffff,ffff0000,00000000 [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. [repeated 15x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO comm 0xf130850 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0 [repeated 15x across cluster]
(RayWorkerWrapper pid=260, ip=172.29.130.200) deepseek-784474d684-52f6w-gpu-group-worker-czczb:260:260 [4] NCCL INFO Channel 23/24 : 0 1 [repeated 170x across cluster]
(RayWorkerWrapper pid=2126) deepseek-784474d684-52f6w-head-nmjfd:2126:2126 [5] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 [4] -1/-1/-1->1->0 [5] -1/-1/-1->1->0 [6] 0/-1/-1->1->-1 [7] 0/-1/-1->1->-1 [8] 0/-1/-1->1->-1 [9] 0/-1/-1->1->-1 [10] 0/-1/-1->1->-1 [11] 0/-1/-1->1->-1 [12] -1/-1/-1->1->0 [13] -1/-1/-1->1->0 [14] -1/-1/-1->1->0 [15] -1/-1/-1->1->0 [16] -1/-1/-1->1->0 [17] -1/-1/-1->1->0 [18] 0/-1/-1->1->-1 [19] 0/-1/-1->1->-1 [20] 0/-1/-1->1->-1 [21] 0/-1/-1->1->-1 [22] 0/-1/-1->1->-1 [23] 0/-1/-1->1->-1 [repeated 13x across cluster]
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:2123 [6] NCCL INFO P2P Chunksize set to 524288 [repeated 15x across cluster]
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:23049 [6] NCCL INFO [Proxy Service UDS] Device 6 CPU core 74 [repeated 15x across cluster]
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:23048 [6] NCCL INFO [Proxy Service] Device 6 CPU core 72 [repeated 15x across cluster]
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:2123 [6] NCCL INFO Channel 23/0 : 0[6] -> 1[7] via P2P/IPC [repeated 274x across cluster]
(RayWorkerWrapper pid=2122) deepseek-784474d684-52f6w-head-nmjfd:2122:2122 [3] NCCL IN [repeated 2x across cluster]
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:2123 [6] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] -1/-1/-1->0->1 [7] -1/-1/-1->0->1 [8] -1/-1/-1->0->1 [9] -
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:2123 [6] NCCL INFO Check P2P Type intraNodeP2pSupport 1 directMode 0 [repeated 7x across cluster]
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:2123 [6] NCCL INFO Connected all rings, use ring PXN 0 GDR 1 [repeated 7x across cluster]
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:2123 [6] NCCL INFO Connected all trees [repeated 7x across cluster]
(RayWorkerWrapper pid=2104) INFO 08-22 01:16:45 [custom_all_reduce_utils.py:208] generating GPU P2P access cache in /root/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(RayWorkerWrapper pid=260, ip=172.29.130.200) deepseek-784474d684-52f6w-gpu-group-worker-czczb:260:1458 [4] NCCL INFO [Proxy Progress]
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:23061 [6] NCCL INFO [Proxy Progress] Device 6 CPU core 83 [repeated 3x across cluster]
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:2123 [6] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512 [repeated 3x across cluster]
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:2123 [6] NCCL INFO 24 coll channels, 24 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer [repeated 3x across cluster]
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:2123 [6] NCCL INFO CC Off, workFifoBytes 1048576 [repeated 3x across cluster]
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:2123 [6] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin. [repeated 3x across cluster]
(RayWorkerWrapper pid=2123) deepseek-784474d684-52f6w-head-nmjfd:2123:2123 [6] NCCL INFO ncclCommInitRank comm 0xf13ac10 rank 0 nranks 2 cudaDev 6 nvmlDev 6 busId ba000 commId 0x8972330c449483df - Init COMPLETE [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] Error executing method 'init_device'. This might cause deadlock in distributed execution. [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] Traceback (most recent call last): [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 611, in execute_method [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] return run_method(self, method, args, kwargs) [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 6x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2985, in run_method [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] return func(*args, **kwargs) [repeated 6x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^ [repeated 6x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/ray/util/tracing/tracing_helper.py", line 461, in _resume_span [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] return method(self, *_args, **_kwargs) [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 181, in init_device [repeated 6x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] self.worker.init_device() # type: ignore [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] init_worker_distributed_environment(self.vllm_config, self.rank, [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 581, in init_worker_distributed_environment [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] ensure_model_parallel_initialized(parallel_config.tensor_parallel_size, [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1121, in ensure_model_parallel_initialized [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] initialize_model_parallel(tensor_model_parallel_size, [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1063, in initialize_model_parallel [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] _TP = init_model_parallel_group(group_ranks, [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 858, in init_model_parallel_group [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] return GroupCoordinator( [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^ [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce.py", line 146, in __init__ [repeated 9x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] self.device_communicator = device_comm_cls( [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^ [repeated 6x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] self.ca_comm = CustomAllreduce( [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] if not current_platform.is_rocm() and not _can_p2p(rank, world_size): [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/custom_all_reduce_utils.py", line 245, in gpu_p2p_access_check [repeated 6x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] if not gpu_p2p_access_check(rank, i): [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] get_world_group().barrier() [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/torch/distributed/distributed_c10d.py", line 4640, in barrier [repeated 6x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] torch.distributed.barrier(group=self.cpu_group) [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] File "/usr/local/lib/python3.12/dist-packages/torch/distributed/c10d_logger.py", line 81, in wrapper [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] work.wait() [repeated 3x across cluster]
(RayWorkerWrapper pid=2126) ERROR 08-22 01:16:56 [worker_base.py:619] RuntimeError: [/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [172.29.160.14]:36537 [repeated 3x across cluster]
Traceback (most recent call last):
File "/usr/local/bin/vllm", line 10, in
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 54, in main
args.dispatch_function(args)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 52, in cmd
uvloop.run(run_server(args))
File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1791, in run_server
await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1811, in run_server_worker
async with build_async_engine_client(args, client_config) as engine_client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 158, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 194, in build_async_engine_client_from_engine_args
async_llm = AsyncLLM.from_vllm_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 163, in from_vllm_config
return cls(
^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 117, in __init__
self.engine_core = EngineCoreClient.make_async_mp_client(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 98, in make_async_mp_client
return AsyncMPClient(*client_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 677, in __init__
super().__init__(
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 408, in __init__
with launch_core_engines(vllm_config, executor_class,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
next(self.gen)
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 697, in launch_core_engines
wait_for_engine_startup(
File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 750, in wait_for_engine_startup
raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

```

### Steps to Reproduce

```
apiVersion: orchestration.aibrix.ai/v1alpha1
kind: RayClusterFleet
metadata:
name: deepseek
labels:
app.kubernetes.io/name: aibrix
app.kubernetes.io/managed-by: kustomize
spec:
replicas: 4
selector:
matchLabels:
model.aibrix.ai/name: deepseek
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
template:
metadata:
labels:
model.aibrix.ai/name: deepseek
annotations:
ray.io/overwrite-container-cmd: "true"
spec:
rayVersion: "2.10.0"
headGroupSpec:
rayStartParams:
dashboard-host: "0.0.0.0"
template:
metadata:
labels:
model.aibrix.ai/name: deepseek
spec:
volumes:
- hostPath:
path: /raid/DeepSeek-V3
type: Directory
name: model-path
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
containers:
- name: ray-head
image: vllm/vllm-openai:v0.10.0
command: ["/bin/bash", "-c"]
args:
- >
ulimit -n 65536 &&
apt update && apt install -y wget net-tools && pip3 install ray[default] pyarrow pandas &&
echo "[INFO] Starting Ray head node..." &&
echo "$KUBERAY_GEN_RAY_START_CMD" &&
eval "$KUBERAY_GEN_RAY_START_CMD" &

echo "[INFO] Waiting for Ray dashboard to be ready..." &&
until curl --max-time 5 --fail http://127.0.0.1:8265 > /dev/null 2>&1; do
echo "[WAITING] $(date -u +'%Y-%m-%dT%H:%M:%SZ') - Ray dashboard not ready yet...";
sleep 2;
done &&
echo "[SUCCESS] Ray dashboard is available!" &&

vllm serve /data/model \
--served-model-name deepseek \
--tensor-parallel-size 2 \
--pipeline-parallel-size 8 \
--gpu-memory-utilization 0.9 \
--distributed-executor-backend ray \
--host 0.0.0.0 \
--port 8000 \
--kv-transfer-config '{"kv_connector":"AIBrixOffloadingConnectorV1Type1", "kv_role":"kv_both"}'
ports:
- containerPort: 6379
name: gcs-server
- containerPort: 8265
name: dashboard
- containerPort: 10001
name: client
- containerPort: 8000
name: service
env:
- name: NCCL_NET_PLUGIN
value: none
- name: NCCL_DEBUG
value: "INFO"
- name: NCCL_NVLS_ENABLE
value: "0"
- name: VLLM_USE_V1
value: "1"
- name: AIBRIX_KV_CACHE_OL_L1_CACHE_ENABLED
value: "0"
- name: AIBRIX_KV_CACHE_OL_L2_CACHE_BACKEND
value: "infinistore"
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_CONNECTION_TYPE
value: "RDMA"
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_IB_PORT
value: "1"
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_LINK_TYPE
value: "IB"
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_VISIBLE_DEV_LIST
value: "mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7"
- name: AIBRIX_KV_CACHE_OL_META_SERVICE_BACKEND
value: "redis"
- name: AIBRIX_KV_CACHE_OL_META_SERVICE_URL
value: "redis://kvcache-cluster-redis:6379"
- name: AIBRIX_KV_CACHE_OL_META_SERVICE_CLUSTER_META_KEY
value: "kvcache_nodes"
- name: VLLM_RPC_TIMEOUT
value: "1000000"
resources:
limits:
nvidia.com/gpu: 8
rdma/15b3_1021_0: 1
rdma/15b3_1021_1: 1
rdma/15b3_1021_2: 1
rdma/15b3_1021_3: 1
rdma/15b3_1021_4: 1
rdma/15b3_1021_5: 1
rdma/15b3_1021_6: 1
rdma/15b3_1021_7: 1
requests:
nvidia.com/gpu: 8
rdma/15b3_1021_0: 1
rdma/15b3_1021_1: 1
rdma/15b3_1021_2: 1
rdma/15b3_1021_3: 1
rdma/15b3_1021_4: 1
rdma/15b3_1021_5: 1
rdma/15b3_1021_6: 1
rdma/15b3_1021_7: 1
volumeMounts:
- mountPath: /data/model
name: model-path
- mountPath: /dev/shm
name: dshm
securityContext:
capabilities:
add: [ "IPC_LOCK" ]
- name: aibrix-runtime
image: aibrix/runtime:v0.3.0
command:
- aibrix_runtime
- --port
- "8080"
env:
- name: INFERENCE_ENGINE
value: vllm
- name: INFERENCE_ENGINE_ENDPOINT
value: http://localhost:8000
- name: PYTORCH_CUDA_ALLOC_CONF
value: "expandable_segments:True"
ports:
- containerPort: 8080
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 30
periodSeconds: 3
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 50
periodSeconds: 5
resources:
limits:
cpu: "1"
requests:
cpu: "1"
workerGroupSpecs:
- groupName: gpu-group
replicas: 1
minReplicas: 1
maxReplicas: 1
rayStartParams: {}
template:
metadata:
labels:
model.aibrix.ai/name: deepseek
spec:
volumes:
- hostPath:
path: /raid/DeepSeek-V3
type: Directory
name: model-path
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
containers:
- name: ray-worker
image: vllm/vllm-openai:v0.10.0
env:
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NCCL_NET_PLUGIN
value: none
- name: NCCL_DEBUG
value: "INFO"
- name: NCCL_NVLS_ENABLE
value: "0"
command: [ "/bin/bash", "-c" ]
args:
- >
ulimit -n 65536 &&
echo "$KUBERAY_GEN_RAY_START_CMD" &&
echo "$MY_POD_IP" &&
eval "$KUBERAY_GEN_RAY_START_CMD --node-ip-address=$MY_POD_IP" &&
tail -f /dev/null
lifecycle:
preStop:
exec:
command: [ "/bin/sh", "-c", "ray stop" ]
resources:
limits:
nvidia.com/gpu: 8
rdma/15b3_1021_0: 1
rdma/15b3_1021_1: 1
rdma/15b3_1021_2: 1
rdma/15b3_1021_3: 1
rdma/15b3_1021_4: 1
rdma/15b3_1021_5: 1
rdma/15b3_1021_6: 1
rdma/15b3_1021_7: 1
requests:
nvidia.com/gpu: 8
rdma/15b3_1021_0: 1
rdma/15b3_1021_1: 1
rdma/15b3_1021_2: 1
rdma/15b3_1021_3: 1
rdma/15b3_1021_4: 1
rdma/15b3_1021_5: 1
rdma/15b3_1021_6: 1
rdma/15b3_1021_7: 1
volumeMounts:
- mountPath: /data/model
name: model-path
- mountPath: /dev/shm
name: dshm
securityContext:
capabilities:
add: [ "IPC_LOCK" ]

---

apiVersion: v1
kind: Service
metadata:
name: deepseek
labels:
model.aibrix.ai/name: deepseek
prometheus-discovery: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
spec:
selector:
model.aibrix.ai/name: deepseek
ray.io/node-type: head
ports:
- name: serve
port: 8000
protocol: TCP
targetPort: 8000
- name: http
port: 8080
protocol: TCP
targetPort: 8080

---

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
name: allow-aibrix-gateway-to-access-services-route
namespace: default
spec:
from:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespace: aibrix-system
to:
- group: ""
kind: Service

---

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: deepseek-router
namespace: aibrix-system
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: aibrix-eg
namespace: aibrix-system
rules:
- backendRefs:
- group: ""
kind: Service
name: deepseek
namespace: default
port: 8000 # or 8000 if you're not using the runtime sidecar
weight: 1
matches:
- headers:
- name: model
type: Exact
value: deepseek
path:
type: PathPrefix
value: /v1/completions
- headers:
- name: model
type: Exact
value: deepseek
path:
type: PathPrefix
value: /v1/chat/completions
timeouts:
request: 120s


---
apiVersion: orchestration.aibrix.ai/v1alpha1
kind: KVCache
metadata:
name: kvcache-cluster
namespace: default
annotations:
kvcache.orchestration.aibrix.ai/backend: infinistore
infinistore.kvcache.orchestration.aibrix.ai/link-type: "IB"
spec:
metadata:
redis:
runtime:
image: aibrix-cn-beijing.cr.volces.com/aibrix/redis:7.4.2
replicas: 1
resources:
requests:
cpu: 1000m
memory: 1Gi
limits:
cpu: 1000m
memory: 1Gi
service:
type: ClusterIP
ports:
- name: service
port: 12345
targetPort: 12345
protocol: TCP
- name: admin
port: 8088
targetPort: 8088
protocol: TCP
watcher:
image: aibrix-cn-beijing.cr.volces.com/aibrix/kvcache-watcher:v0.3.0
imagePullPolicy: Always
resources:
requests:
cpu: "500m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "256Mi"
cache:
replicas: 2
template:
spec:
containers:
- name: cache
image: aibrix-cn-beijing.cr.volces.com/aibrix/infinistore:v0.2.42-20250506
imagePullPolicy: IfNotPresent
securityContext:
capabilities:
add:
- IPC_LOCK
- SYS_RESOURCE
resources:
requests:
cpu: "10000m"
memory: "120Gi"
rdma/15b3_1021_0: 1
rdma/15b3_1021_1: 1
rdma/15b3_1021_2: 1
rdma/15b3_1021_3: 1
rdma/15b3_1021_4: 1
rdma/15b3_1021_5: 1
rdma/15b3_1021_6: 1
rdma/15b3_1021_7: 1
limits:
cpu: "10000m"
memory: "120Gi"
rdma/15b3_1021_0: 1
rdma/15b3_1021_1: 1
rdma/15b3_1021_2: 1
rdma/15b3_1021_3: 1
rdma/15b3_1021_4: 1
rdma/15b3_1021_5: 1
rdma/15b3_1021_6: 1
rdma/15b3_1021_7: 1
```

### Expected behavior

run rayclusterfleet normally with l2 cache

### Environment

v0.4.0

Start with the L2 cache example in the linked kvcache-offloading documentation and compare it with the RayClusterFleet setup that produced the head-pod logs. Trace the reported vLLM initialization failure around gpu_p2p_access_check and Ray worker startup; done means establishing whether the configuration is supported and documenting or fixing the failure so the cluster starts successfully.

Written by the indexing model from the issue text.