vllm-project / vllm-project/aibrix
[vineyard]vllm kv cache can not store into vineyard memory
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 697
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 104
Description
### 🐛 Describe the bug
When testing the model inference,vllm log has printed vineyard_llm_cache.py updated kv cache data successfully in aibrix/vllm-openai contianer. But kv cache was not written to vineyard memory . The vineyard memory is not increased either.
```shell
DEBUG 05-07 02:31:46 vineyard_llm_cache.py:694] update kv cache: #prefix=3952, #tokens=16, updated=16
(VllmWorkerProcess pid=375269) DEBUG 05-07 02:31:46 vineyard_llm_cache.py:694] update kv cache: #prefix=3952, #tokens=16, updated=16
(VllmWorkerProcess pid=375270) DEBUG 05-07 02:31:46 vineyard_llm_cache.py:694] update kv cache: #prefix=3952, #tokens=16, updated=16
(VllmWorkerProcess pid=375270) DEBUG 05-07 02:31:46 vineyard_llm_cache.py:646] prefetch_kv_caches: matched={116: 495}
(VllmWorkerProcess pid=375268) DEBUG 05-07 02:31:46 vineyard_llm_cache.py:646] prefetch_kv_caches: matched={116: 495}
DEBUG 05-07 02:31:46 vineyard_llm_cache.py:646] prefetch_kv_caches: matched={116: 495}
(VllmWorkerProcess pid=375269) DEBUG 05-07 02:31:46 vineyard_llm_cache.py:646] prefetch_kv_caches: matched={116: 495}
DEBUG 05-07 02:31:46 vineyard_llm_cache.py:694] update kv cache: #prefix=4448, #tokens=16, updated=16
(VllmWorkerProcess pid=375270) DEBUG 05-07 02:31:46 vineyard_llm_cache.py:694] update kv cache: #prefix=4448, #tokens=16, updated=16
(VllmWorkerProcess pid=375268) DEBUG 05-07 02:31:46 vineyard_llm_cache.py:694] update kv cache: #prefix=4448, #tokens=16, updated=16
(VllmWorkerProcess pid=375269) DEBUG 05-07 02:31:46 vineyard_llm_cache.py:694] update kv cache: #prefix=4448, #tokens=16, updated=16
(VllmWorkerProcess pid=375268) DEBUG 05-07 02:31:46 vineyard_llm_cache.py:646] prefetch_kv_caches: matched={116: 495}
DEBUG 05-07 02:31:46 vineyard_llm_cache.py:646] prefetch_kv_caches: matched={116: 495}
```
Even after deleted the deepseek-coder-7b-kvcache Pod and the vineyard process, the VLLM logs still show that vineyard_llm_cache.py has successfully updated the kv cache. There is no "Failed to connect to vineyard" exception reported in the logs.
How can make the kv cache be stored in vineyard memory? Even though the vineyard process no longer exists, why the VLLM logs still print vineyard_llm_cache.py has successfully updated the kv cache?
### Steps to Reproduce
apiVersion: apps/v1
kind: Deployment
metadata:
name: deepseek-coder-7b-instruct
labels:
model.aibrix.ai/name: deepseek-coder-7b-instruct
model.aibrix.ai/port: "8000"
spec:
replicas: 1
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
selector:
matchLabels:
model.aibrix.ai/name: deepseek-coder-7b-instruct
template:
metadata:
labels:
model.aibrix.ai/name: deepseek-coder-7b-instruct
spec:
containers:
- name: vllm-openai
image: aibrix/vllm-openai:v0.6.1-edb07092-20250118
imagePullPolicy: Always
command:
- python3
- -m
- vllm.entrypoints.openai.api_server
- --port
- "8000"
- --uvicorn-log-level
- warning
- --model
- deepseek-ai/deepseek-coder-6.7b-instruct
- --served-model-name
- deepseek-coder-7b-instruct
- --max-model-len
- "8192" # please modify this field if your gpu has more room
- --enable-prefix-caching
- --disable-fastapi-docs
env:
- name: VLLM_USE_VINEYARD_CACHE
value: "1"
- name: VINEYARD_CACHE_CPU_MEM_LIMIT_GB
value: "10"
- name: AIBRIX_LLM_KV_CACHE
value: "1"
- name: AIBRIX_LLM_KV_CACHE_KV_CACHE_NS
value: "aibrix"
- name: AIBRIX_LLM_KV_CACHE_CHUNK_SIZE
value: "16"
- name: AIBRIX_LLM_KV_CACHE_SOCKET
value: /var/run/vineyard.sock
- name: AIBRIX_LLM_KV_CACHE_RPC_ENDPOINT
value: "deepseek-coder-7b-kvcache-rpc:9600"
- name: VINEYARD_CACHE_ENABLE_ASYNC_UPDATE
value: "1"
- name: "VINEYARD_CACHE_METRICS_ENABLED"
value: "1"
volumeMounts:
- mountPath: /var/run
name: kvcache-socket
resources:
limits:
nvidia.com/gpu: "1"
requests:
nvidia.com/gpu: "1"
volumes:
- name: kvcache-socket
hostPath:
path: /var/run/vineyard-kubernetes/default/deepseek-coder-7b-kvcache
---
apiVersion: v1
kind: Service
metadata:
labels:
model.aibrix.ai/name: deepseek-coder-7b-instruct
prometheus-discovery: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
name: deepseek-coder-7b-instruct # Note: The Service name must match the label value `model.aibrix.ai/name` in the Deployment
namespace: default
spec:
ports:
- name: serve
port: 8000
protocol: TCP
targetPort: 8000
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
model.aibrix.ai/name: deepseek-coder-7b-instruct
type: ClusterIP
### Expected behavior
no crash in the beginning and no restart should be required to run the engine successfully.
### Environment
AIbrix version 0.2.0
vllm image version: aibrix/vllm-openai:v0.6.1-edb07092-20250118
Contributor guide
Research direction
Start with vineyard_llm_cache.py and the supplied Kubernetes Deployment, Service, environment variables, socket mount, and RPC endpoint. Reproduce the deployment while checking the Vineyard process, socket, cache logs, and Vineyard memory usage, then compare behavior after deleting the cache Pod and Vineyard process. Done means the KV cache is actually stored in Vineyard memory and the logs accurately reflect the connection state without requiring an engine restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, kubernetes, python
- Domain
- ai-infra-agents, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100