allenai/olmes

View on GitHub

Reproducible, flexible LLM evaluations

Stars: 395
Forks: 105
Open beginner issues: 0
Indexed issues: 24

Dominant language: Python
License: Apache-2.0
Last GitHub push: Mar 24, 2026
Latest indexed: Sep 13, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Open All

24 open issues indexed

Release Plan for IFEval-OOD and HREF Open

allenai/olmes#4 · 9 comments · 6 reactions · 0 assignees · Indexed Sep 13, 2026
Running the benchmark suite on very big models Open

allenai/olmes#5 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
drop score very low Open

allenai/olmes#7 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 13, 2026
"Likelihood" based functions are not implemented for API-based models Open

allenai/olmes#10 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Reproducing MATH numbers from OLMo-2 release blog Open

allenai/olmes#11 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 13, 2026
potential breaking API changes from alpacal_eval? Open

allenai/olmes#13 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
What's the recommended way to test local models? Open

allenai/olmes#21 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 13, 2026
Missing datasets Open

allenai/olmes#25 · 4 comments · 1 reaction · 0 assignees · Indexed Sep 13, 2026
Argument list too long Open

allenai/olmes#26 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Reproduction of Tulu-3-8B-SFT on MMLU Open

allenai/olmes#27 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
GGUF models Open

allenai/olmes#28 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Reproduction of Olmo3 Open

allenai/olmes#30 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Multi-worker evaluation fails: CUDA multiprocessing error Open

allenai/olmes#31 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Unexpected keyword argument 'prompt_token_ids' for Gemma 3 Eval Open

allenai/olmes#32 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
[BUG] alpaca_eval_v2:bpb::tulu yields incorrect, inflated BPB scores due to num_chars defaulting to 1 Open

allenai/olmes#33 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Feature Request: Adaptive Item Selection for Expensive Generative Evaluations Open

allenai/olmes#34 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Dataset 'allenai/aime-2021-2025' doesn't exist on the Hub or cannot be accessed. Open

allenai/olmes#36 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
MT MBPP dataset unavailable Open

allenai/olmes#37 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Consider relaxing python library dependencies to make olmes easier to install Open

allenai/olmes#38 · 0 comments · 3 reactions · 0 assignees · Indexed Sep 13, 2026
What's the recommended (or any) way to run OLMES on multiple GPUs? Open

allenai/olmes#39 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
Documentation - List of working dataset pipelines Open

allenai/olmes#46 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
newer transformers library versions break olmo3, fix provided in a PR Open

allenai/olmes#54 · 2 comments · 1 reaction · 0 assignees · Indexed Sep 13, 2026
Optional EvalPort export for OLMES task instances and predictions Open

allenai/olmes#58 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026
OpenAI-compat: --model-type litellm with openai/ prefix and api_base_url /v1 Open

allenai/olmes#59 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 13, 2026

allenai/olmes

Get new issues in your inbox