mlcommons / mlcommons/modelbench
Add skeleton 1.0 benchmark
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 134
- Forks
- 36
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 17
Description
The skeleton should be a basic minimum structure that lets us run something. It should incorporate whatever we can easily get from current workstream outputs, and provide a solid structure for adding things on. Along the way, let's make a list of what we'll need to get to a proper 1.0.
Here's about where we'll start:
- at least 3 prompts
- synthetic prompts from workstream 3?
- not ground truth prompts
- at least one hazard from workstream 1's definitions
- 1 test per hazard
- llama guard 2 to start (or whatever's coming from workstream 2)
- hazard score is fraction unsafe
- personas are all combined
- benchmark scoring: use same reference models and approach as in 0.5, but in separate code
- benchmarks are separated by language and persona? Start with english and normal-ish persona
This from WS1 looks helpful: https://docs.google.com/document/d/1JUQyd4UvpXlduqaMyIdCnyGF9uleu8ZyB07PX9ZUB-k/edit#heading=h.2ua223rrefyo
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing 0.5 benchmark implementation and the outputs or definitions from workstreams 1–3. Shape the 1.0 skeleton around the listed prompts, hazard tests, Llama Guard 2, scoring, and English normal-persona scope. Done means a runnable benchmark plus a documented list of requirements for a proper 1.0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100