[32B Dense] Mid-training downstream evals

One-time setup

Clone oellm-autoexp and check out the branch with the scripts (until it is merged):

git clone git@github.com:OpenEuroLLM/oellm-autoexp.git
cd oellm-autoexp && git checkout feat/downstream-eval-32b
git submodule update --init submodules/oellm-eval
uv tool install -p 3.12 submodules/oellm-eval 

oellm-eval is a submodule pinned on branch feat/vllm-data-parallel; the scripts import it as a
Python package. Set EVAL_REPO=<path> only to test your own oellm-eval checkout instead.

You also need membership of the Slurm account you want to charge (e-ext-2025e02-108 or
e-sta-openeurollm, chosen with ACCOUNT=), and write access to the shared eval directory.

Weight storage

Converted HF checkpoints can be found here:

• /e/fscratch/e-sta-openeurollm/jj1/eval/hf
• /e/scratch/e-sta-openeurollm/production_training/downstream_eval_32b/hf