huggingface / huggingface/lighteval
Add single `mmlu` config for `lighteval` suite
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Currently it seems that to run MMLU with the `lighteval` suite, one needs to specify all the subsets individually as is done for leaderboard task set [here](https://github.com/huggingface/lighteval/blob/6a3e3b9245db9a596d636d71424aa0f2e4358ff6/tasks_examples/open_llm_leaderboard_tasks.txt#L7-L64).
Is it possible to group these together so that one can just run something like this:
```shell
accelerate launch --multi_gpu --num_processes=8 run_evals_accelerate.py \
--tasks="lighteval|mmlu|5|0" \
--model_args "pretrained=Qwen/Qwen1.5-0.5B-Chat" \
--output_dir "./scratch/evals/" --override_batch_size 1
```
Or do you recommend using one of the other suites like `helm` or `original` for this task?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.