facebookresearch / facebookresearch/fairseq2

Create fairseq2 LM Evaluation Harness Wrapper

Open
#707 0 comments 1 reaction 0 assignees View on GitHub
enhancement recipe
Dominant language
Python
Stars
1.1k
Forks
144
Avg merge
4d 1h
Merged PRs (30d)
1

Description

**Describe the solution you would like:**
Create a fairseq2 wrapper class/script to enable LM Evaluation Harness, https://github.com/EleutherAI/lm-evaluation-harness. This

We need to create a [wrapper class](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md#external-library-usage) that implements the following functions:
- `loglikelihood()`
- `loglikelihood_rolling()`
- `generate_until()`

Example wrapper classes/scripts from other frameworks that integrates LM Evaluation Harness:
- TorchTune's [`eleuther_eval.py`](https://github.com/pytorch/torchtune/blob/main/recipes/eleuther_eval.py): the `_EvalWrapper` class
- GPTFast's [`eval.py`](https://github.com/pytorch-labs/gpt-fast/blob/main/eval.py): this one has a script with command line interface

They are both single classes/scripts to port.

**Describe the alternatives you have considered:**
An alternative is to convert fairseq2 model to HuggingFace format and then evaluating on LM Evaluation Harness codebase. But this is a lengthy process.

With the proposed solution, we can evaluate all of LM Evaluation Harness tasks (and future tasks as well!) naturally from within fairseq2.

**Additional Context:**
LM Evaluation Harness, https://github.com/EleutherAI/lm-evaluation-harness, is a widely used tool to evaluate language models on a wide range of tasks.
Although it is mainly uses HuggingFace `transformers`, it can use any model from any framework, provided that we create a wrapper class as described above.

You can see in their README that they already support other frameworks such as NVIDIA nemo, vLLM, OpenAI, JAX, Megatron DeepSpeed, etc.

**Stretch Goals:**
In the future we can even support [BigCode Evaluation Harness](https://github.com/bigcode-project/bigcode-evaluation-harness) that evaluates LLMs on coding tasks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.