vllm-project / vllm-project/aibrix

kv cache deploy the model across different GPUs, it create two etcd pod!

Open
#910 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### 🚀 Feature Description and Motivation

Heterogeneous GPU test `kv cache` , and it create two etcd.

### Use Case

I deploy two deployment on two machine, one in a100, another in t4.I create two kv-cache with different name.The T4 pod the following error occurred, and single t4 pod work well.
```
kind: KVCache
metadata:
name: deepseek-coder-7b-kvcache-t4
namespace: prdsafe
annotations:
kvcache.orchestration.aibrix.ai/pod-affinity-workload: deepseek-coder-7b-instruct
kvcache.orchestration.aibrix.ai/node-affinity-gpu-type: T4
kvcache.orchestration.aibrix.ai/node-affinity-key: "nvidia.com/gpu.product"
```
```
kind: KVCache
metadata:
name: deepseek-coder-7b-kvcache
namespace: prdsafe
annotations:
kvcache.orchestration.aibrix.ai/pod-affinity-workload: deepseek-coder-7b-instruct
kvcache.orchestration.aibrix.ai/node-affinity-gpu-type: A100
kvcache.orchestration.aibrix.ai/node-affinity-key: "nvidia.com/gpu.product"
```
```
WARNING 03-27 04:30:21 multiproc_gpu_executor.py:56] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 03-27 04:30:21 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 03-27 04:30:21 selector.py:217] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 03-27 04:30:21 selector.py:116] Using XFormers backend.
(VllmWorkerProcess pid=212) INFO 03-27 04:30:21 selector.py:217] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
(VllmWorkerProcess pid=212) INFO 03-27 04:30:21 selector.py:116] Using XFormers backend.
/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_fwd")
(VllmWorkerProcess pid=212) /usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=212) @torch.library.impl_abstract("xformers_flash::flash_fwd")
/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_bwd")
(VllmWorkerProcess pid=212) /usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
(VllmWorkerProcess pid=212) @torch.library.impl_abstract("xformers_flash::flash_bwd")
(VllmWorkerProcess pid=212) Process VllmWorkerProcess:
(VllmWorkerProcess pid=212) Traceback (most recent call last):
(VllmWorkerProcess pid=212) File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(VllmWorkerProcess pid=212) self.run()
(VllmWorkerProcess pid=212) File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(VllmWorkerProcess pid=212) self._target(*self._args, **self._kwargs)
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 210, in _run_worker_process
(VllmWorkerProcess pid=212) worker = worker_factory()
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/vllm/executor/gpu_executor.py", line 25, in create_worker
(VllmWorkerProcess pid=212) wrapper.init_worker(**kwargs)
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 476, in init_worker
(VllmWorkerProcess pid=212) self.worker = worker_class(*args, **kwargs)
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 100, in __init__
(VllmWorkerProcess pid=212) self.model_runner: GPUModelRunnerBase = ModelRunnerClass(
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 965, in __init__
(VllmWorkerProcess pid=212) self.attn_backend = get_attn_backend(
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/vllm/attention/selector.py", line 117, in get_attn_backend
(VllmWorkerProcess pid=212) from vllm.attention.backends.xformers import ( # noqa: F401
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/vllm/attention/backends/xformers.py", line 6, in
(VllmWorkerProcess pid=212) from xformers import ops as xops
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/xformers/ops/__init__.py", line 8, in
(VllmWorkerProcess pid=212) from .fmha import (
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/__init__.py", line 12, in
(VllmWorkerProcess pid=212) from . import ck, ck_decoder, ck_splitk, cutlass, decoder, flash, small_k, triton_splitk
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/xformers/ops/fmha/triton_splitk.py", line 89, in
(VllmWorkerProcess pid=212) if TYPE_CHECKING or _is_triton_available():
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/xformers/__init__.py", line 38, in func_wrapper
(VllmWorkerProcess pid=212) value = func()
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/xformers/__init__.py", line 54, in _is_triton_available
(VllmWorkerProcess pid=212) if torch.cuda.get_device_capability("cuda") < (8, 0):
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 451, in get_device_capability
(VllmWorkerProcess pid=212) prop = get_device_properties(device)
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 465, in get_device_properties
(VllmWorkerProcess pid=212) _lazy_init() # will define _get_device_properties
(VllmWorkerProcess pid=212) File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 300, in _lazy_init
(VllmWorkerProcess pid=212) raise RuntimeError(
(VllmWorkerProcess pid=212) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
ERROR 03-27 04:30:21 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 212 died, exit code: 1
INFO 03-27 04:30:21 multiproc_worker_utils.py:123] Killing local vLLM worker processes
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
Process SpawnProcess-1:
self.run()
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 84, in run
for result in iter(self.result_queue.get, _TERMINATE):
File "/usr/lib/python3.10/multiprocessing/queues.py", line 103, in get
res = self._recv_bytes()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 421, in _recv_bytes
return self._recv(size)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
TypeError: 'NoneType' object cannot be interpreted as an integer
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 169, in _enqueue_task
self._task_queue.put((task_id, method, args, kwargs))
File "/usr/lib/python3.10/multiprocessing/queues.py", line 88, in put
raise ValueError(f"Queue {self!r} is closed")
ValueError: Queue is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/server.py", line 236, in run_rpc_server
server = AsyncEngineRPCServer(async_engine_args, usage_context, rpc_path)
File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/rpc/server.py", line 34, in __init__
self.engine = AsyncLLMEngine.from_engine_args(
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 735, in from_engine_args
engine = cls(
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 615, in __init__
self.engine = self._init_engine(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 835, in _init_engine
return engine_class(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 262, in __init__
super().__init__(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py", line 326, in __init__
self.model_executor = executor_class(
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 222, in __init__
super().__init__(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/distributed_gpu_executor.py", line 27, in __init__
super().__init__(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 47, in __init__
self._init_executor()
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 124, in _init_executor
self._run_workers("init_device")
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 193, in _run_workers
worker_outputs = [
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_gpu_executor.py", line 194, in
worker.execute_method(method, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 176, in execute_method
self._enqueue_task(future, method, args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/multiproc_worker_utils.py", line 172, in _enqueue_task
raise ChildProcessError("worker died") from e
ChildProcessError: worker died
ERROR 03-27 04:30:27 api_server.py:188] RPCServer process died before responding to readiness probe
```

### Proposed Solution

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.