mlcommons / mlcommons/endpoints
feat: multiple performance configs in a submission config
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
Each submission config contains accuracy config and a single performance config. When sweeping through concurrencies, requiring an accuracy run for each concurrency value increases runtime, especially when the accuracy dataset takes similar/longer time compared to the performance dataset. Propose to allow multiple performance configs within a submission config.
Before:
load_pattern:
type: "concurrency"
target_concurrency: 512
After:
load_pattern:
type: "concurrency"
target_concurrency: 64,128,256,512
This allows amortization of doing accuracy runs and encourages using a single endpoint config for different concurrency levels.
Several possible considerations:
- target concurrency values are strictly ascending
- Requests for a new concurrency are issued after requests for the current concurrency value are all received (not issued).
- Accuracy run uses the highest concurrency
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by tracing submission config handling and the load_pattern execution path, then inspect how accuracy and performance runs are coordinated. Done means a submission config accepts multiple target concurrency values and follows the stated ordering, request-draining, and highest-concurrency accuracy considerations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100