Unified evaluation pipeline

Create a standardized evaluation procedure for testing our and other models. Eg, a release like this https://www.reddit.com/r/LocalLLaMA/comments/1nbi95c/tilde_ai_releases_tildeopen_llm_an_opensource/ https://huggingface.co/TildeAI/TildeOpen-30b cannot be properly checked (in this particular case there are no evals available by releasing body, only some perplexity scores that tell almost nothing).

I imagine evals we did for open-sci-ref can be a good start. The aim should be that for any model of interest, we should be able to create a plot like that putting the model into the same reference frame across scales where other already evaluated models are visible.This will allow for quick and fair estimation where our own models or other releases stand

We take the already automated eval procedure.

https://github.com/OpenEuroLLM/oellm-cli

with plotting code here

https://github.com/OpenEuroLLM/notebooks/tree/main/plot_results_release

Aim is a tool that given a model provides full table of its evals together with other models and the plot

Currently, references only exist for smaller scales. We should eval more existing reference models on >= 7B.

On longer term, we should incorporate evals which Joaquin is working on
Also on longer term, we should consider using https://github.com/mlfoundations/evalchemy/ to incorporate reasoning evals in addition (See the already existing task https://github.com/orgs/OpenEuroLLM/projects/4?pane=issue&itemId=127602530&issue=OpenEuroLLM%7Coellm-cli%7C19)