mlcommons / mlcommons/endpoints
Low concurrency timeouts
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
Concurrency=1 requires the most amount of time for any hardware not specialized for single concurrency use.
We would like to keep the concurrency=1 (or lower concurrency in general for instance between 1 and 8) supported via the system, but that may require us to tweak the constraints.
One proposal is to use a sampled (similarly distributed to the original dataset) subset of the original dataset for lower concurrency runs. This would require comparisons to be only made between the lower concurrency points on the pareto.
Alternatively, we can limit the execution time, but that would mean that random subset of the dataset get processed within the time limit, which makes comparison unfair - even between lower concurrency submissions due to different random subsets.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Start by tracing how concurrency constraints, dataset selection, and execution limits are implemented; define a fair policy for concurrency values from 1 to 8, with comparisons limited to compatible Pareto points, and validate that lower-concurrency runs remain comparable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100