huggingface / huggingface/lighteval

Add single `mmlu` config for `lighteval` suite

Open
#61 2 comments 0 reactions 0 assignees View on GitHub
feature science-team
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

Currently it seems that to run MMLU with the `lighteval` suite, one needs to specify all the subsets individually as is done for leaderboard task set [here](https://github.com/huggingface/lighteval/blob/6a3e3b9245db9a596d636d71424aa0f2e4358ff6/tasks_examples/open_llm_leaderboard_tasks.txt#L7-L64).

Is it possible to group these together so that one can just run something like this:

```shell
accelerate launch --multi_gpu --num_processes=8 run_evals_accelerate.py \
--tasks="lighteval|mmlu|5|0" \
--model_args "pretrained=Qwen/Qwen1.5-0.5B-Chat" \
--output_dir "./scratch/evals/" --override_batch_size 1
```

Or do you recommend using one of the other suites like `helm` or `original` for this task?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.