huggingface / huggingface/lighteval

[BUG] concurrent_requests in litellm backend limited to 100

Open
#1,100 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
I am evaluating a model deployed with vllm locally and I want to increase the `concurrent_requests` to 128 to speed up the evaluation. However, it seems there are some limitation and the number of concurrent requests sent to the vllm server can not surpass 100. I am sure that the deployed vllm server can handle 128 concurrent request, so I wondering if there are any configs that I missed in `lighteval` or `litellm`?

## To Reproduce
My config:
```
model_parameters:
provider: "hosted_vllm"
model_name: "hosted_vllm/qwen"
base_url: "http://localhost:8000/v1"
api_key: ""
timeout: 100000000
concurrent_requests: 128
max_model_length: 38912
generation_parameters:
temperature: 0.6
top_p: 0.9
top_k: 20
seed: 42
```
My conmand:
```lighteval endpoint litellm lighteval_vllm_config.yaml $DATASET --output-dir ./results --save-details```

## Expected behavior
The vllm server can receive 128 concurrent requests.

## Version info
lighteval version: 0.13.0
litellm version: 1.80.8

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.