vllm-project / vllm-project/aibrix

Engine exception asyncio.exceptions.CancelledError in benchmark

Open
#1,773 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### 🐛 Describe the bug

* prefillers throw the following exception.
* The prefillers and decoders are still able to work in the 2nd test.

Image

### Steps to Reproduce

1. Deploy vllm 2p2d tp2 for Qwen3-32B.
2. Allocate 1 RDMA nic per pod.
3. Schedule all pods onto the same node.
4. Conduct benchmarking:

```
TOKENIZER="/data01/models/Qwen3-32B"
MODEL="qwen3-32b"
HOST="${LB_EXTERNAL_IP}"
PORT="80"
REQ_RATE=1.0
INPUT_LEN=8000
OUTPUT_LEN=200

python3 benchmark_serving.py --port $PORT --host $HOST --seed $(date +%s) \
--model $MODEL \
--tokenizer $TOKENIZER \
--dataset-name random --random-input-len ${INPUT_LEN} --random-output-len ${OUTPUT_LEN} \
--num-prompts 200 --burstiness 100 --request-rate ${REQ_RATE} --metric-percentiles 95 \
--backend openai-chat --endpoint /v1/chat/completions --routing-strategy "pd" --ignore-eos
```

### Expected behavior

No error reported during benchmarking.

### Environment

* AIBrix 0.5.0
* VKE
* Node conf: 8GPUs, 8RDMA

Contributor guide

Open the contributing guide

Research direction

Start with the reported benchmark_serving.py command and reproduce the failure using the listed AIBrix 0.5.0, VKE, Qwen3-32B, 2p2d tp2, and RDMA setup. Trace where asyncio.exceptions.CancelledError is raised during the prefillers' first test and confirm completion when benchmarking no longer reports the exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.