mlcommons / mlcommons/endpoints
BFCL scoring/reporting & robustness cleanup (live_relevance, no-metric PASS, drain cap, plot guard, dead code)
Open
@Palanivelg is already working on this.
Since Jul 22, 2026.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
From PR #346 review (2026-07-02). Grouped low/medium cleanup items.
bfcl_v4_scorer.py:204—live_relevanceinverted semantics published. It takes the_score_astpath withground_truth="[]", awarding credit for NOT calling; BFCL relevance is the opposite (correct when the model DOES call). It's excluded from category aggregates via_UNSCORED_SUBSETS, but the backwards value is still emitted underresults.json"unscored_subsets". Either score with relevance semantics (1.0 if has_calls) or drop it from output.compliance/checker.py:187— silent PASS when no accuracy metric applies. If no_ACCURACY_METRIC_KEYSkey intersects the model'sgolden_accuracy,check_accuracyreturns zeroaccuracy:*checks andComplianceReport.passedreports PASS on config-lock alone. "No applicable accuracy metric" should be an explicit FAIL/warning.load_generator/session.py:380—stop_current_phasedoesn't set_drain_event. If themax_duration_mscap fires while the perf phase is already in its drain wait, the callback is a no-op; withdrain.performance_timeout_s: nulla hung request keeps the run alive indefinitely despite the cap.metrics/results_plots.py:358— one-directional length guard.counts/bucketsare extracted independently with no length validation; a shorterhist_countsmakesaxes[1].bar(...)raise a shape-mismatchValueError, andgenerate_plotsdoesn't catch it, aborting plotting for the whole report dir. Validate both arrays to a common length.dataset_manager/predefined/bfcl_v4/multi_turn.py:120— dead/duplicated plumbing.get_tools_for_turn/excluded_functionunused;MULTI_TURN_SUBSETSduplicated with the package__init__.py;DEFAULT_MAX_STEPS_PER_TURN=25duplicated inbfcl_v4_execution.py:47andbfcl_v4_multi_turn_runner.py:40. Risk of silent drift.openai/openai_msgspec_adapter.py:88— dead guard.ModelParams.max_new_tokensis non-optional with a default, sois not Noneis always true; the omission path is unreachable. Make the fieldint | Noneor drop the guard.
Deferred out of PR #346.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.