huggingface / huggingface/lighteval

[FT] Add regression tests to improve library stability with vLLM backend

Open
#872 0 comments 0 reactions 0 assignees View on GitHub
prio science-team
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Issue encountered

There are currently no integrations tests for the vLLM backend which means the following often happens:

1. Bumping `vllm` to a new version breaks the `lighteval` API
2. Bumping `lighteval` introduces unintended regressions due to refactors or changes to the public API

In addition, many prompt functions are not unit tested, so the expected inputs / outputs are only verified when users report issues in the downstream evaluation metrics.

## Solution/Feature

All of the above points can be addressed by having end-to-end regression tests that run on multiple GPUs to test:

* Regressions on DP
* Regressions on TP
* Regressions on benchmark datasets, metrics, etc

For post-training benchmarks, I suggest we use the `Qwen/Qwen3-0.6B` model as the reference since it is fast to run and supports hybrid reasoning via `/think` and `/no_think` system prompts.

I've created a table here for the most critical evals: https://docs.google.com/spreadsheets/d/1He7zlSBNv2EiQ9hwpMhGNFzFG1A8E7XvPG3d5yvcBNA/edit?usp=sharing

I'll let @loubnabnl @hynky1999 @guipenedo add the corresponding core evals for pre-training.

The basic idea is to run these core evals on a daily basis (with both DP and TP) so that regressions can be caught before end users are affected.

Also note, that to catch potential issues with future `vllm`, `transformers` releases etc we should add a separate set of integration tests against the `main` branch of those projects. Here is how we do it in TRL: https://github.com/huggingface/trl/blob/main/.github/workflows/tests_latest.yml

## Possible alternatives

N/A

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.