mlcommons / mlcommons/modelbench
Usable journal of benchmark run events (includes machine readability)
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 134
- Forks
- 36
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 17
Description
Output is a JSON or JSONL file that has:
- metadata
- run environment
- code
- libraries
- data sources used
- start and end times
- errors
- all errors that cause prompts to fail
- any lower-level errors and failures
- every prompt
- with unique id
- every response
- with the prompt id
- and cache status
- every annotation
- with the prompt id
- and cache status
- statistics
- cache usage
Mostly this will be log-like, in that entries will be in time order. Correlation of related things (like putting together a prompt, an annotation, and responses) will be done with post-processing.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file, test, or entry point is named. Start by locating where benchmark runs emit prompts, responses, annotations, errors, and statistics, then determine how run metadata is gathered. Done means producing a time-ordered JSON or JSONL journal containing the listed records and fields, with identifiers and cache status sufficient for post-processing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100