mlcommons / mlcommons/endpoints
Parameterize run-timeout coverage across benchmark lifecycle stages
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
Context
PR #409 adds integration coverage for settings.timeouts.run_timeout_s during service launch, live performance, and metrics drain. Coverage is not yet organized as one sharp lifecycle-stage matrix equivalent to the SIGINT integration coverage.
Keep this as a test-focused follow-up rather than expanding PR #409 further. That PR is already approximately 1,510 non-test changed lines against the 1,300-line review target; this work would not add production behavior, but it would increase an already large review.
Lifecycle matrix
Parameterize the same run-watchdog contract across:
- Service startup.
- Endpoint-worker startup/connect.
- Live session execution.
- In-flight response drain.
- Metrics tokenization drain.
Accuracy scoring/finalization is intentionally excluded and tracked separately in #464.
Acceptance criteria
- Each stage deterministically blocks at a known lifecycle boundary before the watchdog fires.
- Every case surfaces
ExecutionErrorattributed to the run timeout. - No case yields a
complete: truerun-level report. - Child services and endpoint workers are not leaked.
- Overlapping one-off timeout tests are removed or folded into the parameterized matrix.
- Test synchronization uses observable stage boundaries rather than arbitrary sleeps.
- No production timeout behavior changes.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the integration coverage added by PR #409 and compare its lifecycle handling with the SIGINT integration coverage. Build the parameterized matrix for the five listed stages, using observable boundaries, then verify timeout errors, incomplete reports, and cleanup without changing production behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100