mlcommons / mlcommons/endpoints

Parameterize run-timeout coverage across benchmark lifecycle stages

Open
#465 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
21
Forks
28
Avg merge
3d 17h
Merged PRs (30d)
13

Description

Context

PR #409 adds integration coverage for settings.timeouts.run_timeout_s during service launch, live performance, and metrics drain. Coverage is not yet organized as one sharp lifecycle-stage matrix equivalent to the SIGINT integration coverage.

Keep this as a test-focused follow-up rather than expanding PR #409 further. That PR is already approximately 1,510 non-test changed lines against the 1,300-line review target; this work would not add production behavior, but it would increase an already large review.

Lifecycle matrix

Parameterize the same run-watchdog contract across:

  1. Service startup.
  2. Endpoint-worker startup/connect.
  3. Live session execution.
  4. In-flight response drain.
  5. Metrics tokenization drain.

Accuracy scoring/finalization is intentionally excluded and tracked separately in #464.

Acceptance criteria

  • Each stage deterministically blocks at a known lifecycle boundary before the watchdog fires.
  • Every case surfaces ExecutionError attributed to the run timeout.
  • No case yields a complete: true run-level report.
  • Child services and endpoint workers are not leaked.
  • Overlapping one-off timeout tests are removed or folded into the parameterized matrix.
  • Test synchronization uses observable stage boundaries rather than arbitrary sleeps.
  • No production timeout behavior changes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the integration coverage added by PR #409 and compare its lifecycle handling with the SIGINT integration coverage. Build the parameterized matrix for the five listed stages, using observable boundaries, then verify timeout errors, incomplete reports, and cleanup without changing production behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.