vllm-project / vllm-project/production-stack
bug: json.decoder.JSONDecodeError
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.6k
- Forks
- 503
- Avg merge
- 4d 17h
- Merged PRs (30d)
- 8
Description
after deploying the router server in the k8s environment, i run the stress test by locust but the error below occured at the line.
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 992 (char 991)
when i request to the vllm instance directly, it works well.
In addition, when i request to the router only once (not cocurrently), it works well too.
so i think the router is the problem.
i want to know how to receive many cocurrent requests through vllm-router like vllm independent instance.
is there any arguments or required settings for this?
my enviornment is:
nvidia H100 x 8 GPU Server
RHEL 8.8
kubenetes: v1.31.1
vllm-router 0.1.2.dev24+g168f937.d20250413
vllm: v0.7.3 / v0.8.3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting src/vllm_router/services/request_service/request.py at lines 164-166, then reproduce the failure with concurrent Locust requests against the router and compare responses with direct vLLM requests. Determine why the router receives or parses an unterminated JSON response and verify that concurrent requests complete without JSONDecodeError under the reported Kubernetes environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100