mlcommons / mlcommons/inference
mixtral-8x7b: Reference Implementation Accuracy Failure on H200
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
When running reference implementation on H200, I see an accuracy failure:
| Metric | Target Score | H200 Reference Implementation | Percentage Diff |
|---|---|---|---|
| rouge1 | 45.5989 | 45.127 | 1.034893386 |
| rouge2 | 23.3526 | 22.9785 | 1.601962951 |
| rougeL | 30.4608 | 30.4806 | 0.065001576 |
| gsm8k | 73.66 | 74.06 | 0.543035569 |
| mbxp | 60.16 | 60.22 | 0.099734043 |
| tokens per sample | 144.84 | 283.5 | 95.73322287 |
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Mixtral-8x7B reference implementation on H200 and compare its ROUGE, GSM8K, MBXP, and tokens-per-sample results with the reported targets. Trace the benchmark configuration and reference implementation to identify the accuracy and throughput discrepancy; done means the H200 results meet the stated targets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100