huggingface / huggingface/lighteval

Add Code-Centric Interface to LightEval for Enhanced Usability

Open
#148 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

Enhancing the functionality of LightEval to better accommodate coding workflows is paramount. The current approach relies heavily on command-line interaction (CLI), but a more code-centric interface would greatly benefit users.

Consider the following refinement:

```python
# Install LightEval package
pip install lighteval

from lighteval import Evaluator, EvaluatorArguments

def configure_dataset():
# Define dataset formatting and evaluation parameters here

# Initialize evaluator for custom dataset evaluations
evaluator = Evaluator(
model=model,
eval_dataset=dataset,
metric="loglikelihood_acc",
dataset_text_field=configure_dataset,
args=EvaluatorArguments(
# Specify additional arguments for evaluation configuration
# e.g., batch size, evaluation steps, etc.
# Example:
batch_size=32,
num_workers=4,
...
),
)

# Initiate the evaluation process
evaluator.evaluate()

# Display results and publish statistics to the Hugging Face Hub
evaluator.show_results()
evaluator.push_results()
```

This revised approach emphasizes a more structured and Pythonic usage of LightEval, with clear functions to define dataset formatting and evaluation specifics. Additionally, it leverages the `EvaluatorArguments` class to encapsulate additional evaluation configurations like batch size and number of workers. The usage of `Evaluator` and related methods is aligned with conventional Python programming paradigms, enhancing usability and integration within code-centric workflows.

if this is a feature you guys believe would be beneficial, I am eager to contribute to its development and enhancement.

@clefourrier @NathanHB

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.