mlcommons / mlcommons/endpoints
Performance discrepancy between submission config and performance-only config
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
When running submission config (accuracy + performance), accuracy samples are issued right after performance samples, causing accuracy samples to be batched with the last few performance samples. This results in reduced performance using accuracy+performance config especially under high concurrency. Attaching config files used for testing and results. Note that the lack of tokens/s report and "failed" samples are separate issues.
Propose to issue accuracy samples after results from all performance samples have been received.
offline_llama3_1b_cnn_full.yaml
report_1b_full.txt
offline_llama3_1b_cnn_perf.yaml
report_1b_perf.txt
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the attached offline_llama3_1b_cnn_full.yaml and offline_llama3_1b_cnn_perf.yaml configurations with their report files to reproduce the discrepancy under high concurrency. Trace submission scheduling and verify that accuracy samples are issued only after results from all performance samples have been received, while treating the tokens/s and failed-sample issues as separate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100