mlcommons / mlcommons/inference
Querying intermediate results
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
I am running MLPerf Inference datacenter suite on a CPU only device following the instructions on the documentation.
The suggested sample size/query counts seem to take a very long time to reach completion.
- Would it be possible to query intermediate results (such as throughput) when the benchmark is executing?
- How are the sample sizes correlated with the accuracy of results? For instance, does llama2 CPU run need the same sample count (24576) as GPU? This is suggested here
I see the following prints on my terminal, but I am not sure how to interpret these results:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked MLPerf Inference Llama 2 70B documentation and the llama2-70b offline instructions. Review how the benchmark reports progress during execution and how the documented sample count relates to CPU and GPU runs; done should be a decided approach for intermediate metrics and clear guidance on sample-size interpretation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100