huggingface / huggingface/lighteval

[FT] Support setting the multiple random seeds and decoupling it from model deployment?

Open
#1,110 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Issue encountered
When I want to run the results of multiple seeds for a task, the existing solution only allows specifying the seed during the initialization of the Pipeline (when initializing VLLMModelConfig). However, for evaluating multiple seeds, it is often only necessary to specify the seed during inference. Currently, this solution requires redeployment every time a seed is specified, and the model deployment of VLLM takes a lot of extra time.

## Solution/Feature
One solution is to decouple the setting of the seed from model deployment, allowing for a single deployment to perform multiple evaluations with multiple seeds and calculate the average. This is similar to a multi-task approach.

## Possible alternatives
No other solutions have been thought of.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.