vllm-project / vllm-project/aibrix

Infinistore failed to work in the scale testing

Open
#1,102 10 comments 0 reactions 1 assignee Claimed by @DwyaneShi View on GitHub
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### 🐛 Describe the bug

![Image](https://github.com/user-attachments/assets/ed4e186a-b6bd-4d81-b01b-2964767eaf03)

![Image](https://github.com/user-attachments/assets/83b2a604-a65b-49f6-b4a8-22ca0cfbea16)

### Steps to Reproduce

8 vllm instances, 960 GB Infinistore single instance

### Expected behavior

it should work as expected

### Environment

- AIBrix version: v0.3.0-rc.1

```
apiVersion: v1
kind: Pod
metadata:
name: infinistore
namespace: default
labels:
app: infinistore
spec:
hostNetwork: true
containers:
- name: infinistore
image: aibrix-cn-beijing.cr.volces.com/aibrix/infinistore:20250506
command: ["infinistore"]
args:
- "--manage-port=8088"
- "--dev-name=mlx5_0" # if mlx5_0 is not available, it will fallbacks to other devs
- "--service-port=12345"
- "--link-type=Ethernet"
# - "--hint-gid-index=7"
- "--log-level=debug"
- "--prealloc-size=950"
ports:
- containerPort: 8088
- containerPort: 12345
securityContext:
privileged: true
resources:
requests:
cpu: "10"
memory: "960Gi"
limits:
cpu: "10"
memory: "960Gi"
restartPolicy: Never

```

```
env:
- name: VLLM_RPC_TIMEOUT
value: "1000000"
- name: VLLM_USE_V1
value: "0"
- name: AIBRIX_KV_CACHE_OL_L1_CACHE_ENABLED
value: "0"
- name: AIBRIX_KV_CACHE_OL_L2_CACHE_BACKEND
value: "infinistore"
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_HOST_ADDR
value: "192.168.3.35"
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_SERVICE_PORT
value: "12345"
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_CONNECTION_TYPE
value: "RDMA"
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_IB_PORT
value: "1"
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_LINK_TYPE
value: "Ethernet"
- name: AIBRIX_KV_CACHE_OL_INFINISTORE_VISIBLE_DEV_LIST
value: "mlx5_1:7,mlx5_2:7,mlx5_3:7,mlx5_4:7"
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.