mlcommons / mlcommons/inference
Submission checker logging Whisper performance with wrong metric
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
The 'mlperf_log_summary.txt' for Whisper displays both 'Samples per second' and 'Tokens per second'. 'Tokens per second' is the metric used in the final dashboard. However, the submission checker selects 'Samples per second' in its log output.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the submission checker code that selects the metric for Whisper and the generation of mlperf_log_summary.txt. Verify how the checker chooses between Samples per second and Tokens per second, then reproduce the current log output. Done means the checker reports the metric used by the final dashboard for Whisper.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100