mlcommons / mlcommons/endpoints
Add an overall deadline for the complete compliance audit sequence
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
Context
When audit: is enabled, the CLI runs the normal benchmark first (including finalize_benchmark() and accuracy scoring), then calls run_audit(). audit.only: true skips the preceding main benchmark.
PR #409 gives each AuditRunSpec subrun its own setup-inclusive run_timeout_s deadline so audit subruns match normal benchmark-run semantics. That is not an overall audit budget: a reference phase and fixed-sample phase can each consume the full value, and AuditTest.verify() runs afterward outside those per-subrun deadlines.
origin/main has no active run watchdog for audits; it only applies the existing response/metrics drain budgets inside each subrun.
Scope
Add a first-class overall audit deadline in a follow-up MR rather than overloading the per-benchmark run_timeout_s contract in PR #409. Define its ownership alongside phase configuration work in #449 and scoring/finalization cancellation work in #464.
Acceptance criteria
- One deadline begins before the first audit subrun setup and is shared across every planned audit subrun.
- Remaining budget, rather than a fresh full budget, is passed to each subrun.
- The contract explicitly covers or boundary-checks
AuditTest.verify()and result writing. audit.only: trueuses the same audit deadline semantics.- Expiry produces a non-zero exit, honest incomplete/interrupted artifacts, and no leaked children.
- Existing per-subrun response drain, metrics drain, and interrupted teardown-grace semantics remain independently configurable.
- Integration coverage blocks deterministically during multiple audit lifecycle stages and proves the overall budget is not reset.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading PR #409 and the related phase-configuration and scoring/finalization work in #449 and #464, then trace run_audit(), AuditRunSpec subruns, AuditTest.verify(), and result writing. Done means one shared deadline covers setup, all subruns, verification, and artifacts for both normal and audit-only runs, with deterministic integration coverage and no leaked children.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100