mlcommons / mlcommons/endpoints
Consolidate run-outcome state flags into one abort/outcome model
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
Follow-up from PR #409 review (raised by @roborluo in https://github.com/mlcommons/endpoints/pull/409#discussion_r3823849649).
The run orchestrator tracks its outcome through many scattered booleans and derived fields — watchdog.fired, SigintGovernor.interrupted/.forced, session.stop_requested, session_completed_normally, BenchmarkResult.run_timed_out/.user_interrupted, Report.state/.complete — combined at several layers (_run_benchmark_async, finalize_benchmark, run_benchmark, run_audit). Each flag is individually justified, but the combinations are hard to review and easy to get subtly wrong.
Proposal: a single run-outcome enum/struct (e.g. RunOutcome: COMPLETED | ABORTED_USER | ABORTED_FORCED | ABORTED_TIMEOUT | ABORTED_EXTERNAL | INCOMPLETE_METRICS) owned by the orchestrator, derived once, consumed everywhere (artifact rewriting, exit-code mapping, audit certification).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the run-outcome handling in _run_benchmark_async, finalize_benchmark, run_benchmark, and run_audit, then review PR #409 and its discussion. Trace how the listed flags and derived fields affect artifact rewriting, exit-code mapping, and audit certification. Done means one orchestrator-owned outcome model is derived once and consumed consistently across those paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100