mlcommons / mlcommons/endpoints

Add a deadline for accuracy scoring and finalization

Open
#464 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
21
Forks
28
Avg merge
3d 17h
Merged PRs (30d)
13

Description

Context

PR #409 consolidates the existing benchmark timeout configuration. accuracy_drain_timeout_s only bounds outstanding endpoint responses from the accuracy load phase. It does not cover score_accuracy() or finalization, and scorers with SKIP_ENDPOINT_PHASE bypass that response drain entirely.

The run watchdog is currently cancelled before finalize_benchmark(). External-service scorers such as SWE-bench and subprocess-based scorers such as VBench/LiveCodeBench can therefore wait without a benchmark-level deadline.

Scope

Add an explicit deadline for accuracy scoring and finalization in a follow-up MR. Define how it composes with run_timeout_s rather than silently changing the existing response-drain timeout.

Acceptance criteria

  • Scoring/finalization has an explicit, documented deadline.
  • External-service and subprocess scorers have defined cancellation/termination behavior.
  • SKIP_ENDPOINT_PHASE scorers are covered.
  • Expiry produces honest incomplete/interrupted artifacts and a non-zero exit.
  • No scorer process or service is leaked after expiry.
  • Integration coverage exercises service-backed and subprocess-backed timeout paths.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading PR #409 and tracing the run watchdog, score_accuracy(), finalize_benchmark(), accuracy_drain_timeout_s, and SKIP_ENDPOINT_PHASE paths. Map how run_timeout_s currently composes with scoring and finalization, then add integration coverage for service-backed and subprocess-backed scorers; done means expiry leaves honest incomplete artifacts, returns non-zero, and leaks no scorer process or service.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.