mlfoundations/evalchemy

View on GitHub

Automatic evals for LLMs

Stars: 610
Forks: 87
Open beginner issues: 0
Indexed issues: 23

Dominant language: HTML
License: No license data
Last GitHub push: Dec 23, 2025
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Beginner friendly Everything

Open All

23 open issues indexed

OpenAI-compat: --model openai-chat-completions with full chat URL, not curator or vLLM Open

Difficulty 2/5 1-3 hours Newbie friendliness 48/100

mlfoundations/evalchemy#161 · Opened Aug 28, 2026
[bug] GPQA-Diamond ground truth answers are all D Open

Difficulty 2/5 1-3 hours Newbie friendliness 45/100

mlfoundations/evalchemy#155 · 2 comments · Opened Feb 24, 2026
vLLM backend crashes with "please provide at least one prompt" when max_gen_toks exceeds model context window Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

mlfoundations/evalchemy#152 · Opened Jan 3, 2026
[bug] LiveCodeBenchv5 Open

Difficulty 2/5 1-3 hours Newbie friendliness 50/100

mlfoundations/evalchemy#149 · Opened Dec 23, 2025
[Bug] google/codegemma-7b-it Evaluation Results All Zero Open

Difficulty 4/5 3-5 days Newbie friendliness 30/100

mlfoundations/evalchemy#147 · Opened Dec 16, 2025
Only return loose result in IFEval? Open

Difficulty 2/5 1-3 hours Newbie friendliness 58/100

mlfoundations/evalchemy#145 · Opened Oct 28, 2025
final answer extraction in reasoning model's output Open

Difficulty 4/5 3-5 days Newbie friendliness 20/100

mlfoundations/evalchemy#144 · Opened Oct 2, 2025
LM Eval Tasks not found Open

Difficulty 3/5 1-2 days Newbie friendliness 42/100

mlfoundations/evalchemy#143 · Opened Sep 18, 2025
Difference between self.max_tokens self.max_new_tokens in eval_instruct.py Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

mlfoundations/evalchemy#142 · Opened Sep 3, 2025
Add logical reasoning task tests? (e.g., Enigmata data) Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mlfoundations/evalchemy#141 · Opened Aug 29, 2025
Support for SWEBench Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

mlfoundations/evalchemy#140 · 1 comment · Opened Aug 12, 2025
Request to add Debug mode Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

mlfoundations/evalchemy#139 · 1 comment · 1 reaction · Opened Jul 23, 2025
per prompt performance and public data Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

mlfoundations/evalchemy#132 · Opened Jun 27, 2025
LCB benchmarks are duplicated Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

mlfoundations/evalchemy#129 · 1 reaction · Opened Jun 13, 2025
How many responses are sampled per query? Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

mlfoundations/evalchemy#105 · 1 comment · Opened Apr 8, 2025
How to change model's temperature ? Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

mlfoundations/evalchemy#99 · 2 comments · 4 reactions · Opened Mar 25, 2025
A suggestion for evalchemy about LiveCodeBench Open

Difficulty 4/5 3-5 days Newbie friendliness 38/100

mlfoundations/evalchemy#96 · 2 comments · Opened Mar 20, 2025
Log samples to huggingface Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

mlfoundations/evalchemy#89 · Opened Mar 5, 2025
How to log model response? Open

Difficulty 2/5 1-3 hours Newbie friendliness 30/100

mlfoundations/evalchemy#84 · 4 comments · Opened Feb 21, 2025
Support multi-node evaluation with vLLM Open

Difficulty 5/5 Over a week Newbie friendliness 28/100

mlfoundations/evalchemy#46 · Opened Dec 28, 2024
lm-evaluation-harness dependency conflicts Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mlfoundations/evalchemy#45 · Opened Dec 21, 2024
Race condition when running alpaca eval? Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mlfoundations/evalchemy#31 · 1 reaction · Opened Nov 23, 2024
Integrate Berkeley Function Calling Leaderboard into Evalchemy Open

mlfoundations/evalchemy#28 · 2 comments · 2 reactions · 1 assignee · Opened Nov 21, 2024

mlfoundations/evalchemy

Get new issues in your inbox