mlcommons / mlcommons/endpoints
Add a deadline for accuracy scoring and finalization
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
Context
PR #409 consolidates the existing benchmark timeout configuration. accuracy_drain_timeout_s only bounds outstanding endpoint responses from the accuracy load phase. It does not cover score_accuracy() or finalization, and scorers with SKIP_ENDPOINT_PHASE bypass that response drain entirely.
The run watchdog is currently cancelled before finalize_benchmark(). External-service scorers such as SWE-bench and subprocess-based scorers such as VBench/LiveCodeBench can therefore wait without a benchmark-level deadline.
Scope
Add an explicit deadline for accuracy scoring and finalization in a follow-up MR. Define how it composes with run_timeout_s rather than silently changing the existing response-drain timeout.
Acceptance criteria
- Scoring/finalization has an explicit, documented deadline.
- External-service and subprocess scorers have defined cancellation/termination behavior.
SKIP_ENDPOINT_PHASEscorers are covered.- Expiry produces honest incomplete/interrupted artifacts and a non-zero exit.
- No scorer process or service is leaked after expiry.
- Integration coverage exercises service-backed and subprocess-backed timeout paths.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading PR #409 and tracing the run watchdog, score_accuracy(), finalize_benchmark(), accuracy_drain_timeout_s, and SKIP_ENDPOINT_PHASE paths. Map how run_timeout_s currently composes with scoring and finalization, then add integration coverage for service-backed and subprocess-backed scorers; done means expiry leaves honest incomplete artifacts, returns non-zero, and leaks no scorer process or service.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100