OpenEuroLLM / OpenEuroLLM/Taskboard

Support OpenJury & Evalchemy in evals

Open
#80 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

T5.1 - static evals T5.2 - dynamic evals
Dominant language
No language data
Stars
3
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Add support in https://github.com/OpenEuroLLM/multi-cluster-eval to run evaluations from https://github.com/mlfoundations/evalchemy/

In particular, we should make sure that at least Alpaca-Eval with a local model judge served with vllm works, e.g. to make sure that we have a good workflow to run this:

python -m eval.eval \
    --model vllm \
    --tasks alpaca_eval \
    --model_args "pretrained=meta-llama/Meta-Llama-3-8B-Instruct" \
    --batch_size 16 \
    --output_path logs

(we may want to start by just exposing a clear way to do this with the output paths specified from environment variables rather than having a CLI to schedule such evals)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the multi-cluster-eval repository and the evalchemy invocation shown in the issue, including how evaluation output paths are handled. Verify the Alpaca-Eval workflow with a local model judge served by vLLM; done means the evaluation can run with output paths supplied through environment variables or a clearly documented scheduling interface.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.