mlcommons / mlcommons/endpoints
compliance: scope edge-agentic checker (rename to edge_agentic.py, required ruleset/model args)
Open
@Palanivelg is already working on this.
Since Jul 22, 2026.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
Follow-up to PR #346 (thread https://github.com/mlcommons/endpoints/pull/346#discussion_r3532061377), flagged by @viraatc as fine to address in a follow-up.
compliance/checker.py is edge-agentic-specific (BFCL metric keys, qwen3.6-27b / mlperf-edge-current defaults, hardcoded attestation note) under a generic top-level compliance/ name.
Proposed:
- Rename to
compliance/edge_agentic.py(or move next to the edge ruleset). - Make
ruleset_name/model_namerequired args instead of defaults. - Name the model's
golden_accuracykeys to match the scorer's breakdown keys (overall_accuracy,normalized_single_turn_score) so the_ACCURACY_METRIC_KEYSmapping disappears. - Generalizing into a real framework can wait until a second benchmark needs one; coordinate with #332.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.