facebookresearch / facebookresearch/fairseq2
Create fairseq2 LM Evaluation Harness Wrapper
- Dominant language
- Python
- Stars
- 1.1k
- Forks
- 144
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
**Describe the solution you would like:**
Create a fairseq2 wrapper class/script to enable LM Evaluation Harness, https://github.com/EleutherAI/lm-evaluation-harness. This
We need to create a [wrapper class](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md#external-library-usage) that implements the following functions:
- `loglikelihood()`
- `loglikelihood_rolling()`
- `generate_until()`
Example wrapper classes/scripts from other frameworks that integrates LM Evaluation Harness:
- TorchTune's [`eleuther_eval.py`](https://github.com/pytorch/torchtune/blob/main/recipes/eleuther_eval.py): the `_EvalWrapper` class
- GPTFast's [`eval.py`](https://github.com/pytorch-labs/gpt-fast/blob/main/eval.py): this one has a script with command line interface
They are both single classes/scripts to port.
**Describe the alternatives you have considered:**
An alternative is to convert fairseq2 model to HuggingFace format and then evaluating on LM Evaluation Harness codebase. But this is a lengthy process.
With the proposed solution, we can evaluate all of LM Evaluation Harness tasks (and future tasks as well!) naturally from within fairseq2.
**Additional Context:**
LM Evaluation Harness, https://github.com/EleutherAI/lm-evaluation-harness, is a widely used tool to evaluate language models on a wide range of tasks.
Although it is mainly uses HuggingFace `transformers`, it can use any model from any framework, provided that we create a wrapper class as described above.
You can see in their README that they already support other frameworks such as NVIDIA nemo, vLLM, OpenAI, JAX, Megatron DeepSpeed, etc.
**Stretch Goals:**
In the future we can even support [BigCode Evaluation Harness](https://github.com/bigcode-project/bigcode-evaluation-harness) that evaluates LLMs on coding tasks.
Contributor guide
Assessment
This issue has not been assessed yet.