mlcommons/ailuminate

View on GitHub

The AILuminate v1.1 benchmark suite is an AI risk assessment benchmark developed with broad involvement from leading AI companies, academia, and civil society.

Stars: 83
Forks: 22
Open beginner issues: 0
Indexed issues: 22

Dominant language: No language data
License: Apache-2.0
Last GitHub push: Jun 11, 2025
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Beginner friendly Everything

Open All

22 open issues indexed

Proposal: Add BFSI hazard domain — FinProof Bench (Banking/Financial Services/Insurance) Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

mlcommons/ailuminate#38 · Opened Sep 13, 2026
Hazard categories are scored on text responses. Physical-AI systems fail in the action space. Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

mlcommons/ailuminate#37 · Opened Sep 1, 2026
Proposal: vulnerable-user reliability overlay for AILuminate (Korean pilot) Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mlcommons/ailuminate#36 · Opened Aug 4, 2026
Refactor top-level information architecture Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mlcommons/ailuminate#30 · Opened Apr 28, 2025
Will test sets in other languages/regions be open source? Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mlcommons/ailuminate#28 · 1 comment · 2 reactions · Opened Apr 11, 2025
Undocumented hazard category: `spc_lgl` Open

Difficulty 1/5 Under an hour Newbie friendliness 35/100

mlcommons/ailuminate#26 · 1 reaction · Opened Mar 25, 2025
How do we obtain evaluation metrics for this demo subset? Open

Difficulty 3/5 1-2 days Newbie friendliness 35/100

mlcommons/ailuminate#23 · 2 comments · Opened Mar 14, 2025
Benchmark the latest high visibility models Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mlcommons/ailuminate#21 · 1 comment · Opened Mar 10, 2025
Benchmark is accessible by the users via an API and integrated onto hyperscale or other 3P platforms (e.g., Hugging Face) Open

mlcommons/ailuminate#20 · 1 assignee · Opened Mar 6, 2025
Rename ModelBench to AILuminate Open

mlcommons/ailuminate#18 · 1 assignee · Opened Feb 27, 2025
Test and publish latest "model of the moment" results: DeepSeek, etc Open

mlcommons/ailuminate#17 · 1 assignee · Opened Feb 27, 2025
Mike documents use cases Open

mlcommons/ailuminate#15 · 1 assignee · Opened Feb 24, 2025
Make it clearer how people can participate Open

mlcommons/ailuminate#11 · 2 comments · 1 assignee · Opened Feb 11, 2025
Make it easier to run A/B tests with SUTs Open

mlcommons/ailuminate#10 · 1 comment · 1 assignee · Opened Feb 11, 2025
Get per-prompt scores from DEMO set Open

mlcommons/ailuminate#9 · 1 comment · 1 assignee · Opened Feb 11, 2025
Make it easier to get access to the official evaluator Open

mlcommons/ailuminate#8 · 1 assignee · Opened Feb 11, 2025
SUT version/time recording (“GPT-4o at 8:53AM Jan 17, 2025) etc. Open

mlcommons/ailuminate#7 · 1 assignee · Opened Feb 11, 2025
Easier onboarding Open

mlcommons/ailuminate#6 · 2 comments · 1 assignee · Opened Feb 11, 2025
Make it easier to do offline asynchronous tests Open

mlcommons/ailuminate#5 · 1 comment · 1 assignee · Opened Feb 11, 2025
Review current docs with Kurt and William Open

mlcommons/ailuminate#4 · 1 comment · 1 assignee · Opened Feb 11, 2025
Clean up ModelBench documentation Open

mlcommons/ailuminate#3 · 1 comment · 1 assignee · Opened Feb 11, 2025
Make it easier to do offline benchmarking Open

Difficulty 5/5 Over a week Newbie friendliness 25/100

mlcommons/ailuminate#2 · 1 comment · Opened Jan 16, 2025

mlcommons/ailuminate

Get new issues in your inbox