huggingface / huggingface/evaluate

Evaluate LLM models like llama/alpaca using evaluate library?

Open
#433 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Hi team, thanks for open source this awesome tool. I am new to the tool and try to ask some questions on LLM evaluation

1. Seems `evaluate` already create some evaluators (Some libs call it tasks I think). Can we use these evaluator for LLM evaluation?
2. I feel different tasks required different datasets. for LLM evaluation, there're popular datasets like MMLU. I am trying to ask Is there tested paring? for example, for QA, I can use dataset1, dataset2 for metric1, metric2 evaluation etc
3. What's the difference between huggingface/evaluate and https://github.com/EleutherAI/lm-evaluation-harness?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.