mlcommons/modelbench

View on GitHub

Run safety benchmarks against AI models and view detailed reports showing how well they performed.

Stars: 134
Forks: 36
Open beginner issues: 0
Indexed issues: 557
Avg merge: 1d 11h
Merged PRs (30d): 17

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 16, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

557 open issues indexed

Why can't we run any NVIDIA NIM suts Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

mlcommons/modelbench#1660 · Opened Sep 14, 2026
Fix mistral SUT model id formatting Open

Difficulty 2/5 1-3 hours Newbie friendliness 68/100

mlcommons/modelbench#1659 · 1 comment · Opened Sep 14, 2026
Misc google SUT failures Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

mlcommons/modelbench#1658 · Opened Sep 14, 2026
Some Google SUTs don't return real content Open

Difficulty 3/5 1-2 days Newbie friendliness 65/100

mlcommons/modelbench#1657 · 2 comments · Opened Sep 14, 2026
Update William's tool to only check models we care about Open

Difficulty 2/5 1-3 hours Newbie friendliness 68/100

mlcommons/modelbench#1656 · Opened Sep 14, 2026
Decide if we should support Google's Interactions API Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

mlcommons/modelbench#1655 · Opened Sep 14, 2026
Fix AWS SUTs Open

Difficulty 3/5 1-2 days Newbie friendliness 48/100

mlcommons/modelbench#1654 · Opened Sep 14, 2026
Anthropic SUT sometimes doesn't return a TextBlock Open

Difficulty 3/5 1-2 days Newbie friendliness 64/100

mlcommons/modelbench#1653 · 1 comment · Opened Sep 14, 2026
Create openrouter adapter Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

mlcommons/modelbench#1648 · 1 comment · Opened Sep 11, 2026
Add default model lists to dynamic SUT providers that don't auto-discover Open

Difficulty 4/5 3-5 days Newbie friendliness 58/100

mlcommons/modelbench#1646 · Opened Sep 9, 2026
Investigate dynamic SUT run failures Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

mlcommons/modelbench#1645 · Opened Sep 9, 2026
Investigate dynamic SUT listing failures Open

Difficulty 4/5 3-5 days Newbie friendliness 42/100

mlcommons/modelbench#1644 · 1 comment · Opened Sep 9, 2026
consistency checker handles new scoring Open

Difficulty 4/5 3-5 days Newbie friendliness 20/100

mlcommons/modelbench#1641 · Opened Sep 3, 2026
Candela server and client support safety benchmark versions Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mlcommons/modelbench#1640 · Opened Sep 3, 2026
Confirm that all results from production run are in metadata Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

mlcommons/modelbench#1638 · Opened Sep 2, 2026
Integrity check modelplane vs production for a couple of riki's SUTs Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

mlcommons/modelbench#1637 · Opened Sep 2, 2026
Consistency checker handles safety v1.2 Open

Difficulty 3/5 1-2 days Newbie friendliness 25/100

mlcommons/modelbench#1633 · Opened Sep 1, 2026
Carry over is_valid annotation from verdict Open

Difficulty 3/5 1-2 days Newbie friendliness 25/100

mlcommons/modelbench#1632 · Opened Aug 31, 2026
Consistency checker crashes with ZeroDivisionError on empty denominators Open

Difficulty 2/5 Half a day Newbie friendliness 84/100

mlcommons/modelbench#1629 · Opened Aug 29, 2026
Integrate Riki's test scorer Open

Difficulty 4/5 3-5 days Newbie friendliness 38/100

mlcommons/modelbench#1628 · Opened Aug 27, 2026
Run Riki's 11 SUTs Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

mlcommons/modelbench#1625 · 2 comments · Opened Aug 26, 2026
DAG embedder uses cloud run host Open

Difficulty 3/5 1-2 days Newbie friendliness 25/100

mlcommons/modelbench#1622 · Opened Aug 24, 2026
Safety 1.2 DAG in Cheval Open

mlcommons/modelbench#1621 · 3 comments · 1 assignee · Opened Aug 24, 2026
Make safety 1.2 DAG based on security 1.1 DAG Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mlcommons/modelbench#1620 · Opened Aug 24, 2026
Design benchmark UIDs Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mlcommons/modelbench#1618 · Opened Aug 21, 2026
Test new safety 1.2 DAG with real test set Open

mlcommons/modelbench#1617 · 1 assignee · Opened Aug 21, 2026
Is a physical-action hazard family in scope for a future AILuminate taxonomy? Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

mlcommons/modelbench#1616 · Opened Aug 21, 2026
Design deployment of sentence embedder. Open

Difficulty 5/5 Over a week Newbie friendliness 20/100

mlcommons/modelbench#1610 · Opened Aug 12, 2026
Replace registries with dynamic factories Open

chore

Difficulty 5/5 Over a week Newbie friendliness 20/100

mlcommons/modelbench#1604 · Opened Aug 6, 2026
Remove Pydantic from TestItems and Prompts Open

chore

Difficulty 3/5 1-2 days Newbie friendliness 35/100

mlcommons/modelbench#1603 · Opened Aug 6, 2026
Gemini SUT works Open

Difficulty 4/5 3-5 days Newbie friendliness 25/100

mlcommons/modelbench#1597 · 1 comment · Opened Aug 3, 2026
GPT SUT works Open

Difficulty 5/5 Over a week Newbie friendliness 15/100

mlcommons/modelbench#1596 · Opened Aug 3, 2026
Claude SUT works Open

Difficulty 5/5 Over a week Newbie friendliness 15/100

mlcommons/modelbench#1595 · Opened Aug 3, 2026
DAG handles n-ary routing Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mlcommons/modelbench#1592 · 1 comment · Opened Jul 30, 2026
Remove old ensemble code from modelgauge Open

chore

Difficulty 3/5 1-2 days Newbie friendliness 48/100

mlcommons/modelbench#1591 · Opened Jul 28, 2026
Pluggable test-item selection strategies for --max-instances (beyond random sampling) Open

Difficulty 5/5 Over a week Newbie friendliness 38/100

mlcommons/modelbench#1589 · 1 comment · Opened Jul 26, 2026
Monitor together dedicated endpoints Open

mlcommons/modelbench#1586 · 2 comments · 2 assignees · Opened Jul 23, 2026
Preference for dedicated endpoints in dynamic HF and Together factories Open

mlcommons/modelbench#1579 · 1 assignee · Opened Jul 20, 2026
Fix broken jailbreak benchmark Open

Difficulty 3/5 1-2 days Newbie friendliness 25/100

mlcommons/modelbench#1578 · Opened Jul 20, 2026
Modelbench respects inputs and accepts UID Open

Difficulty 3/5 1-2 days Newbie friendliness 25/100

mlcommons/modelbench#1577 · Opened Jul 20, 2026
Update together dedicated endpoints to v2.0 Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

mlcommons/modelbench#1576 · Opened Jul 17, 2026
Security running in Candela Open

milestone

Difficulty 5/5 Over a week Newbie friendliness 20/100

mlcommons/modelbench#1574 · Opened Jul 14, 2026
Security (jailbreak) 1.1 benchmark Open

milestone

Difficulty 5/5 Over a week Newbie friendliness 20/100

mlcommons/modelbench#1572 · Opened Jul 13, 2026
Proposal: RL-specific reward-hacking safety benchmark / methodology contribution Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mlcommons/modelbench#1571 · 1 comment · Opened Jul 10, 2026
Investigate huggingface failures / consistency check failures Open

Difficulty 4/5 3-5 days Newbie friendliness 38/100

mlcommons/modelbench#1569 · 1 comment · Opened Jul 2, 2026
Fairer together calling from cheval Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

mlcommons/modelbench#1568 · Opened Jul 2, 2026
Benchmark stalling issues Open

mlcommons/modelbench#1565 · 1 assignee · Opened Jun 30, 2026
Option to run uncalibrated benchmarks Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

mlcommons/modelbench#1564 · 1 comment · Opened Jun 29, 2026
Security tests put extra information in test items for annotator Open

Difficulty 3/5 1-2 days Newbie friendliness 25/100

mlcommons/modelbench#1561 · Opened Jun 25, 2026
Update security benchmark test item construction to include side information Open

mlcommons/modelbench#1560 · 1 assignee · Opened Jun 25, 2026

mlcommons/modelbench

Get new issues in your inbox