mlcommons / mlcommons/endpoints

Add an overall deadline for the complete compliance audit sequence

Open
#466 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
21
Forks
28
Avg merge
3d 17h
Merged PRs (30d)
13

Description

Context

When audit: is enabled, the CLI runs the normal benchmark first (including finalize_benchmark() and accuracy scoring), then calls run_audit(). audit.only: true skips the preceding main benchmark.

PR #409 gives each AuditRunSpec subrun its own setup-inclusive run_timeout_s deadline so audit subruns match normal benchmark-run semantics. That is not an overall audit budget: a reference phase and fixed-sample phase can each consume the full value, and AuditTest.verify() runs afterward outside those per-subrun deadlines.

origin/main has no active run watchdog for audits; it only applies the existing response/metrics drain budgets inside each subrun.

Scope

Add a first-class overall audit deadline in a follow-up MR rather than overloading the per-benchmark run_timeout_s contract in PR #409. Define its ownership alongside phase configuration work in #449 and scoring/finalization cancellation work in #464.

Acceptance criteria

  • One deadline begins before the first audit subrun setup and is shared across every planned audit subrun.
  • Remaining budget, rather than a fresh full budget, is passed to each subrun.
  • The contract explicitly covers or boundary-checks AuditTest.verify() and result writing.
  • audit.only: true uses the same audit deadline semantics.
  • Expiry produces a non-zero exit, honest incomplete/interrupted artifacts, and no leaked children.
  • Existing per-subrun response drain, metrics drain, and interrupted teardown-grace semantics remain independently configurable.
  • Integration coverage blocks deterministically during multiple audit lifecycle stages and proves the overall budget is not reset.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading PR #409 and the related phase-configuration and scoring/finalization work in #449 and #464, then trace run_audit(), AuditRunSpec subruns, AuditTest.verify(), and result writing. Done means one shared deadline covers setup, all subruns, verification, and artifacts for both normal and audit-only runs, with deterministic integration coverage and no leaked children.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.