OpenEuroLLM / OpenEuroLLM/Taskboard

Shared database for evaluations of oellm-cli

Open
#113 0 comments 0 reactions 1 assignee View on GitHub

@MahamadSalah is already working on this.

Since Oct 14, 2025.

T5.1 - static evals
Dominant language
No language data
Stars
3
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Develop a database that allows to access all evaluations done by OELLM on our different clusters.

Once the database is available, it will be used to:

  1. easily share evaluation accross the projects, right now scattered csv/json files are send around
  2. check if an evaluation was already submitted to avoid repeating evaluation multiple times
  3. update dynamically evaluation dashboard like this one which is currently static: https://huggingface.co/spaces/openeurollm/LLM-leaderboard

We should have:

  1. automatic synchronisation of the DB
  2. easy & fast way to access an evaluation (eg the json of a given model on MMLU)
  3. easy & fast way to list evaluations for scientific analysis.

Reference/design doc:

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.