vllm-project / vllm-project/aibrix
Engine exception asyncio.exceptions.CancelledError in benchmark
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
### 🐛 Describe the bug
* prefillers throw the following exception.
* The prefillers and decoders are still able to work in the 2nd test.
### Steps to Reproduce
1. Deploy vllm 2p2d tp2 for Qwen3-32B.
2. Allocate 1 RDMA nic per pod.
3. Schedule all pods onto the same node.
4. Conduct benchmarking:
```
TOKENIZER="/data01/models/Qwen3-32B"
MODEL="qwen3-32b"
HOST="${LB_EXTERNAL_IP}"
PORT="80"
REQ_RATE=1.0
INPUT_LEN=8000
OUTPUT_LEN=200
python3 benchmark_serving.py --port $PORT --host $HOST --seed $(date +%s) \
--model $MODEL \
--tokenizer $TOKENIZER \
--dataset-name random --random-input-len ${INPUT_LEN} --random-output-len ${OUTPUT_LEN} \
--num-prompts 200 --burstiness 100 --request-rate ${REQ_RATE} --metric-percentiles 95 \
--backend openai-chat --endpoint /v1/chat/completions --routing-strategy "pd" --ignore-eos
```
### Expected behavior
No error reported during benchmarking.
### Environment
* AIBrix 0.5.0
* VKE
* Node conf: 8GPUs, 8RDMA
Contributor guide
Research direction
Start with the reported benchmark_serving.py command and reproduce the failure using the listed AIBrix 0.5.0, VKE, Qwen3-32B, 2p2d tp2, and RDMA setup. Trace where asyncio.exceptions.CancelledError is raised during the prefillers' first test and confirm completion when benchmarking no longer reports the exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100