huggingface / huggingface/lighteval

[FT] Upgrade the VLLM dependency to 0.10.2+

Open
#1,002 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Issue encountered
Some newest LLM models can only supported by the new version VLLM. Currently, lighteval only support VLLM >= 0.10.0 and < 0.10.2

## Solution/Feature
I think the main reason to this issue is VLLM >= 0.10.2 use different interface for LLM inference (more specifically, `llm.generate`) . The VLLM >= 0.10.2 doesn't support the keyword argument `prompt_token_ids` in [case1](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L440C37-L440C53) and [case2](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L458)

We can see there are multiple interfaces of `llm.generate` in VLLM==0.10.1.1 in this page: [Link](https://docs.vllm.ai/en/v0.10.1.1/api/vllm/index.html#vllm.LLM.generate), where also indicates `prompt_token_ids` will be deprecated.

Image

However, in the VLLM==0.10.2, there is only one single interface in the official document: [Link](https://docs.vllm.ai/en/v0.10.2/api/vllm/#vllm.LLM.generate), which accept the inputs in `PromptType`. To my best knowledge, the `inputs` in `_generate` [ref.](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L408) function are assumed to be tokenized, so all we need to do is wrap the original `inputs` with `TokensPrompt` [ref.](https://docs.vllm.ai/en/v0.10.2/api/vllm/inputs/data.html#vllm.inputs.data.TokensPrompt) which contains the attribute `prompt_token_ids: list[int]`

**The solution**
1. Insert on line code before [L418](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L418)
```
inputs = [TokensPrompt(prompt_token_ids=input_ids) for input_ids in inputs]
```

2. Remove the keyword argument `prompt_token_ids` at [L440](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L440) and [L458](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/src/lighteval/models/vllm/vllm_model.py#L458C17-L458C33)

3. Update pyproject.toml, to remove `<0.10.2` in [vllm installation](https://github.com/huggingface/lighteval/blob/b1d45e36a6255ebefbfc2aa8999b7b25a007d619/pyproject.toml#L101C1-L101C57)

Done.

## Possible alternatives
A clear and concise description of any alternative solutions or features you've considered.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.