huggingface / huggingface/lighteval
[FT] Upgrade the VLLM dependency to 0.10.2+
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Issue encountered
Some newest LLM models can only supported by the new version VLLM. Currently, lighteval only support VLLM >= 0.10.0 and < 0.10.2
## Solution/Feature
I think the main reason to this issue is VLLM >= 0.10.2 use different interface for LLM inference (more specifically, `llm.generate`) . The VLLM >= 0.10.2 doesn't support the keyword argument `prompt_token_ids` in [case1](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L440C37-L440C53) and [case2](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L458)
We can see there are multiple interfaces of `llm.generate` in VLLM==0.10.1.1 in this page: [Link](https://docs.vllm.ai/en/v0.10.1.1/api/vllm/index.html#vllm.LLM.generate), where also indicates `prompt_token_ids` will be deprecated.
However, in the VLLM==0.10.2, there is only one single interface in the official document: [Link](https://docs.vllm.ai/en/v0.10.2/api/vllm/#vllm.LLM.generate), which accept the inputs in `PromptType`. To my best knowledge, the `inputs` in `_generate` [ref.](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L408) function are assumed to be tokenized, so all we need to do is wrap the original `inputs` with `TokensPrompt` [ref.](https://docs.vllm.ai/en/v0.10.2/api/vllm/inputs/data.html#vllm.inputs.data.TokensPrompt) which contains the attribute `prompt_token_ids: list[int]`
**The solution**
1. Insert on line code before [L418](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L418)
```
inputs = [TokensPrompt(prompt_token_ids=input_ids) for input_ids in inputs]
```
2. Remove the keyword argument `prompt_token_ids` at [L440](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L440) and [L458](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L458C17-L458C33)
3. Update pyproject.toml, to remove `<0.10.2` in [vllm installation](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/pyproject.toml#L101C1-L101C57)
Done.
## Possible alternatives
A clear and concise description of any alternative solutions or features you've considered.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.