Error running llama70b on AIBrix With Distributed KVCache and Cross-Engine KV Reuse

### 🐛 Describe the bug

When I run Llama-70B with AIBrix using Distributed KVCache and Cross-Engine KV Reuse, I get the following error:

[llama3_3_70b_error_log.txt](https://github.com/user-attachments/files/20998985/llama3_3_70b_error_log.txt)

This does not happen when I run a single-GPU model (such as Llama-8B)

### Steps to Reproduce

Deploy AIBrix with the configuration:

```bash
apiVersion: orchestration.aibrix.ai/v1alpha1
kind: KVCache
metadata:
name: kvcache-cluster
annotations:
kvcache.orchestration.aibrix.ai/backend: infinistore
infinistore.kvcache.orchestration.aibrix.ai/link-type: "IB"
infinistore.kvcache.orchestration.aibrix.ai/hint-gid-index: "1"
spec:
metadata:
redis:
runtime:
image: aibrix-cn-beijing.cr.volces.com/aibrix/redis:7.4.2
replicas: 1
resources:
requests:
cpu: 1000m
memory: 1Gi
limits:
cpu: 1000m
memory: 1Gi
service:
type: ClusterIP
ports:
- name: service
port: 12345
targetPort: 12345
protocol: TCP
- name: admin
port: 8088
targetPort: 8088
protocol: TCP
watcher:
image: aibrix-cn-beijing.cr.volces.com/aibrix/kvcache-watcher:v0.3.0
imagePullPolicy: Always
resources:
requests:
cpu: "500m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "256Mi"
cache:
# if template is specified, it will be used to build the cache engine pod.
replicas: 1
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "8000"
prometheus.io/scrape: "true"
spec:
hostNetwork: true
containers:
- name: infinistore
image: aibrix-cn-beijing.cr.volces.com/aibrix/infinistore:v0.2.42-20250506
command: ["infinistore"]
args:
- "--manage-port=8088"
- "--dev-name=ibp0"
- "--service-port=12345"
- "--link-type=IB"
- "--hint-gid-index=1"
- "--log-level=debug"
- "--prealloc-size=80"
# customize your infinistore setting
ports:
- containerPort: 8088
- containerPort: 12345
securityContext:
privileged: true
capabilities:
add:
- CAP_SYS_PTRACE
- IPC_LOCK
- SYS_RESOURCE
resources:
requests:
cpu: "10"
memory: "120Gi"
# vke.volcengine.com/rdma: "1"
limits:
cpu: "10"
memory: "120Gi"
# vke.volcengine.com/rdma: "1"
```

```
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
model.aibrix.ai/name: llama-33-70b-instruct # Note: The label value `model.aibrix.ai/name` here must match with the service name.
model.aibrix.ai/port: "8000"
name: llama-33-70b-instruct
spec:
replicas: 1
selector:
matchLabels:
model.aibrix.ai/name: llama-33-70b-instruct
template:
metadata:
labels:
model.aibrix.ai/name: llama-33-70b-instruct
spec:
hostNetwork: true
volumes:
- name: models
persistentVolumeClaim:
claimName: pvc-models
containers:
- command:
- python3
- -m
- vllm.entrypoints.openai.api_server
- --host
- "0.0.0.0"
- --port
- "8000"
- --uvicorn-log-level
- warning
- --disable-log-requests
- --disable-fastapi-docs
- --swap-space
- "0"
- --no-enable-chunked-prefill
- --model
- meta-llama/Llama-3.3-70B-Instruct
- --served-model-name
# Note: The `--served-model-name` argument value must also match the Service name and the Deployment label `model.aibrix.ai/name`
- llama-33-70b-instruct
- --max-model-len
- "12800"
- --kv-transfer-config
- '{"kv_connector":"AIBrixOffloadingConnector", "kv_role":"kv_both"}'
- --tensor-parallel-size
- "4"
env:
- name: HF_TOKEN
value: {}
- name: VLLM_USE_V1
value: "0"
- name: AIBRIX_KV_CACHE_OL_L1_CACHE_ENABLED
value: "0"
- name: AIBRIX_KV_CACHE_OL_L2_CACHE_BACKEND
value: "infinistore"
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_CONNECTION_TYPE
value: "RDMA"
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_IB_PORT
value: "1"
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_LINK_TYPE
value: "IB"
# mlx_5_1 is the device and 7 is the hinted gid index, if you do not know the gid, you can just type mlx5_1,mlx5_2,...
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_VISIBLE_DEV_LIST
value: "ibp0:1,ibp1:1,ibp2:1,ibp3:1,ibp4:1,ibp5:1,ibp6:1,ibp7:1"
- name: AIBRIX_KV_CACHE_OL_META_SERVICE_BACKEND
value: "redis"
- name: AIBRIX_KV_CACHE_OL_META_SERVICE_URL
value: "redis://kvcache-cluster-redis:6379"
- name: AIBRIX_KV_CACHE_OL_META_SERVICE_CLUSTER_META_KEY
value: "kvcache_nodes"
- name: VLLM_RPC_TIMEOUT
value: "1000000"
image: aibrix-cn-beijing.cr.volces.com/aibrix/vllm-openai:aibrix-kvcache-v0.8.5-20250520
imagePullPolicy: Always
name: vllm-openai
securityContext:
privileged: true
capabilities:
add:
- CAP_SYS_PTRACE
- IPC_LOCK
- SYS_RESOURCE
ports:
- containerPort: 8000
protocol: TCP
resources:
limits:
nvidia.com/gpu: "4"
requests:
nvidia.com/gpu: "4"
volumeMounts:
- name: models
mountPath: /models
livenessProbe:
httpGet:
path: /health
port: 8000
scheme: HTTP
failureThreshold: 3
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
httpGet:
path: /health
port: 8000
scheme: HTTP
failureThreshold: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
startupProbe:
httpGet:
path: /health
port: 8000
scheme: HTTP
failureThreshold: 30
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1

---

apiVersion: v1
kind: Service
metadata:
labels:
model.aibrix.ai/name: llama-33-70b-instruct
prometheus-discovery: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "8080"
name: llama-33-70b-instruct # Note: The Service name must match the label value `model.aibrix.ai/name` in the Deployment
spec:
ports:
- name: serve
port: 8000
protocol: TCP
targetPort: 8000
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
model.aibrix.ai/name: llama-33-70b-instruct
type: ClusterIP
```

### Expected behavior

The llama-33-70b pod should successfully deployed and start serving requests

### Environment

1 * H100