kvcache-ai / kvcache-ai/Mooncake

[Usage]: How should I run disaggregated Prefill-Decode inference with vLLM + Mooncake across two AMD MI308X servers?

Open
#3,061 2 comments 0 reactions 1 assignee Claimed by @tpiperatgod View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Describe your usage question

My goal is to perform disaggregated Prefill-Decode (PD) inference using vLLM together with Mooncake across two AMD MI308X servers.

Initially, I launched vLLM containers on both servers with the following command:
```bash
docker run -it -v /home:/home -v /mnt:/mnt --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --device=/dev/kfd --device=/dev/dri --group-add video --ipc=host --name vllm_alan-mooncake --shm-size 1024G --entrypoint=/bin/bash vllm/vllm-openai-rocm:latest
```
Then I installed Mooncake inside each container.

I plan to start services following the guide here:
https://kvcache-ai.github.io/Mooncake/getting_started/examples/vllm-integration/disagg-prefill-decode.html

The example commands from the documentation are shown below:
### Basic Setup (Different Nodes)
**Prefiller Node (192.168.0.2):**
```bash
vllm serve Qwen/Qwen2.5-7B-Instruct \
--port 8010 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer"}'
```

**Decoder Node (192.168.0.3):**
```bash
vllm serve Qwen/Qwen2.5-7B-Instruct \
--port 8020 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer"}'
```

**Proxy Server:**
```bash
# In vllm root directory.
python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \
--prefiller-host 192.168.0.2 --prefiller-port 8010 \
--decoder-host 192.168.0.3 --decoder-port 8020
```
> NOTE: The Mooncake Connector currently uses the proxy from nixl_integration. This will be replaced with a self-developed proxy in the future.

Requests can then be sent to the proxy server on port 8000.

**Test command:**
```bash
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [
{"role": "user", "content": "Tell me a long story about artificial intelligence."}
]
}'
```

However, I found that the official upstream `vllm/vllm-openai-rocm:latest` Docker image **does not contain this file**:
`tests/v1/kv_connector/nixl_integration/toy_proxy_server.py`

So I have several questions:
1. How should I run disaggregated Prefill-Decode inference with vLLM + Mooncake across two AMD MI308X servers?
2. I located `tests/v1/kv_connector/nixl_integration/toy_proxy_server.py` inside this repository: https://github.com/kvcache-ai/vllm.
Does this mean I need to build and install the customized vLLM fork from **kvcache-ai/vllm source code** to enable Mooncake-based PD disaggregated inference?
3. Is there an official pre-built Docker image available for the kvcache-ai/vllm repository?
4. Am I required to complete the following steps inside my existing vLLM container to get PD disaggregation working:
- Compile and install the customized kvcache-ai/vllm from source
- Reinstall Mooncake afterwards?

Thanks a lot for your guidance!

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.