mlcommons / mlcommons/modelbench
Simplify annotation object hierarchy in modelgauge
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 134
- Forks
- 36
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 17
Description
PR #872 simplified the definition of a TestItem to correspond to only 1 prompt. This opens the door towards further simplification in the annotation object hierarchy as well. I believe TestItemResponseAnnotations, PromptInteractionAnnotations, SUTResponseAnnotations, and SUTCompletionAnnotations can all be merged into one object now.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing PR #872 and locating the definitions and usages of TestItemResponseAnnotations, PromptInteractionAnnotations, SUTResponseAnnotations, and SUTCompletionAnnotations. Trace how the simplified TestItem affects these annotations and identify all references that require updating. Done means the four annotation types are merged into one object without leaving inconsistent hierarchy references.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100