kvcache-ai / kvcache-ai/Mooncake
[Usage]: How should I run disaggregated Prefill-Decode inference with vLLM + Mooncake across two AMD MI308X servers?
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Describe your usage question
My goal is to perform disaggregated Prefill-Decode (PD) inference using vLLM together with Mooncake across two AMD MI308X servers.
Initially, I launched vLLM containers on both servers with the following command:
```bash
docker run -it -v /home:/home -v /mnt:/mnt --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --device=/dev/kfd --device=/dev/dri --group-add video --ipc=host --name vllm_alan-mooncake --shm-size 1024G --entrypoint=/bin/bash vllm/vllm-openai-rocm:latest
```
Then I installed Mooncake inside each container.
I plan to start services following the guide here:
https://kvcache-ai.github.io/Mooncake/getting_started/examples/vllm-integration/disagg-prefill-decode.html
The example commands from the documentation are shown below:
### Basic Setup (Different Nodes)
**Prefiller Node (192.168.0.2):**
```bash
vllm serve Qwen/Qwen2.5-7B-Instruct \
--port 8010 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer"}'
```
**Decoder Node (192.168.0.3):**
```bash
vllm serve Qwen/Qwen2.5-7B-Instruct \
--port 8020 \
--kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer"}'
```
**Proxy Server:**
```bash
# In vllm root directory.
python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \
--prefiller-host 192.168.0.2 --prefiller-port 8010 \
--decoder-host 192.168.0.3 --decoder-port 8020
```
> NOTE: The Mooncake Connector currently uses the proxy from nixl_integration. This will be replaced with a self-developed proxy in the future.
Requests can then be sent to the proxy server on port 8000.
**Test command:**
```bash
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [
{"role": "user", "content": "Tell me a long story about artificial intelligence."}
]
}'
```
However, I found that the official upstream `vllm/vllm-openai-rocm:latest` Docker image **does not contain this file**:
`tests/v1/kv_connector/nixl_integration/toy_proxy_server.py`
So I have several questions:
1. How should I run disaggregated Prefill-Decode inference with vLLM + Mooncake across two AMD MI308X servers?
2. I located `tests/v1/kv_connector/nixl_integration/toy_proxy_server.py` inside this repository: https://github.com/kvcache-ai/vllm.
Does this mean I need to build and install the customized vLLM fork from **kvcache-ai/vllm source code** to enable Mooncake-based PD disaggregated inference?
3. Is there an official pre-built Docker image available for the kvcache-ai/vllm repository?
4. Am I required to complete the following steps inside my existing vLLM container to get PD disaggregation working:
- Compile and install the customized kvcache-ai/vllm from source
- Reinstall Mooncake afterwards?
Thanks a lot for your guidance!
### Before submitting a new issue...
- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
Contributor guide
Assessment
This issue has not been assessed yet.