mlcommons / mlcommons/modelbench

Add skeleton 1.0 benchmark

Open
#398 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
134
Forks
36
Avg merge
1d 11h
Merged PRs (30d)
17

Description

The skeleton should be a basic minimum structure that lets us run something. It should incorporate whatever we can easily get from current workstream outputs, and provide a solid structure for adding things on. Along the way, let's make a list of what we'll need to get to a proper 1.0.

Here's about where we'll start:

  • at least 3 prompts
    • synthetic prompts from workstream 3?
    • not ground truth prompts
  • at least one hazard from workstream 1's definitions
  • 1 test per hazard
  • llama guard 2 to start (or whatever's coming from workstream 2)
  • hazard score is fraction unsafe
  • personas are all combined
  • benchmark scoring: use same reference models and approach as in 0.5, but in separate code
  • benchmarks are separated by language and persona? Start with english and normal-ish persona

This from WS1 looks helpful: https://docs.google.com/document/d/1JUQyd4UvpXlduqaMyIdCnyGF9uleu8ZyB07PX9ZUB-k/edit#heading=h.2ua223rrefyo

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing 0.5 benchmark implementation and the outputs or definitions from workstreams 1–3. Shape the 1.0 skeleton around the listed prompts, hazard tests, Llama Guard 2, scoring, and English normal-persona scope. Done means a runnable benchmark plus a documented list of requirements for a proper 1.0.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.