mlcommons / mlcommons/endpoints

BFCL scoring/reporting & robustness cleanup (live_relevance, no-metric PASS, drain cap, plot guard, dead code)

Open
#393 0 comments 0 reactions 1 assignee View on GitHub

@Palanivelg is already working on this.

Since Jul 22, 2026.

Dominant language
Python
Stars
21
Forks
28
Avg merge
3d 17h
Merged PRs (30d)
13

Description

From PR #346 review (2026-07-02). Grouped low/medium cleanup items.

  • bfcl_v4_scorer.py:204live_relevance inverted semantics published. It takes the _score_ast path with ground_truth="[]", awarding credit for NOT calling; BFCL relevance is the opposite (correct when the model DOES call). It's excluded from category aggregates via _UNSCORED_SUBSETS, but the backwards value is still emitted under results.json "unscored_subsets". Either score with relevance semantics (1.0 if has_calls) or drop it from output.
  • compliance/checker.py:187 — silent PASS when no accuracy metric applies. If no _ACCURACY_METRIC_KEYS key intersects the model's golden_accuracy, check_accuracy returns zero accuracy:* checks and ComplianceReport.passed reports PASS on config-lock alone. "No applicable accuracy metric" should be an explicit FAIL/warning.
  • load_generator/session.py:380stop_current_phase doesn't set _drain_event. If the max_duration_ms cap fires while the perf phase is already in its drain wait, the callback is a no-op; with drain.performance_timeout_s: null a hung request keeps the run alive indefinitely despite the cap.
  • metrics/results_plots.py:358 — one-directional length guard. counts/buckets are extracted independently with no length validation; a shorter hist_counts makes axes[1].bar(...) raise a shape-mismatch ValueError, and generate_plots doesn't catch it, aborting plotting for the whole report dir. Validate both arrays to a common length.
  • dataset_manager/predefined/bfcl_v4/multi_turn.py:120 — dead/duplicated plumbing. get_tools_for_turn/excluded_function unused; MULTI_TURN_SUBSETS duplicated with the package __init__.py; DEFAULT_MAX_STEPS_PER_TURN=25 duplicated in bfcl_v4_execution.py:47 and bfcl_v4_multi_turn_runner.py:40. Risk of silent drift.
  • openai/openai_msgspec_adapter.py:88 — dead guard. ModelParams.max_new_tokens is non-optional with a default, so is not None is always true; the omission path is unreachable. Make the field int | None or drop the guard.

Deferred out of PR #346.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.