mlcommons / mlcommons/endpoints
[Bug]: Inconsistency with the field `min_duration_ms` in config.yaml
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
Bug Description
Issue
For small models and high concurrency numbers the following config snippet causes unexpected behavior:
settings:
runtime:
min_duration_ms: 600000
max_duration_ms: 3600000
Expected
The number of samples is calculated dynamically to satisfy min_duration_ms.
Observed
The experiment is run to produce a performance and/or an accuracy reports, without any time check.
We have seen experiments with Duration: 74.34 seconds in their report.txt files.
Workaround
Explicitly set n_samples_to_issue chosen so, that the minimum time constrain is satisfied:
settings:
runtime:
min_duration_ms: 600000
max_duration_ms: 3600000
n_samples_to_issue: 100000
Steps to Reproduce
- Create a config with big enough
min_duration_ms, say, 10h, but withoutn_samples_to_issue. - Run an experiment with the config.
- Check the time taken (or look into
report.txtfor the duration of the experiment).
Environment
inference-endpoints dad7d1a6462d91cd081cb58a07c51ed54fb0ba5e
Relevant Logs
Before submitting
- I searched existing issues and found no duplicates
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the runtime handling for min_duration_ms, max_duration_ms, and n_samples_to_issue in the configuration path. Reproduce the issue with a large minimum duration, then inspect report.txt and verify that experiments do not finish before the configured minimum duration when n_samples_to_issue is omitted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100