mlcommons / mlcommons/endpoints

BFCL test coverage: BFCLv4MultiTurnScorer.score() and bfcl_v4 generate() selection logic

Open
#394 0 comments 0 reactions 1 assignee View on GitHub

@Palanivelg is already working on this.

Since Jul 22, 2026.

Dominant language
Python
Stars
21
Forks
28
Avg merge
3d 17h
Merged PRs (30d)
13

Description

From PR #346 review (2026-07-02).

  • bfcl_v4_multi_turn_scorer.py:59score() untested. Untested branches: force_terminated early-fail, turn_count_mismatch, the checker_exception catch, valid/invalid bookkeeping, unweighted-mean category aggregation, and the empty-valid_subset_accs -> 0.0 branch. Can be tested by patching multi_turn_checker (same approach as the runner/execution-bridge tests). Also pin the string-typed subset_scores contract (f"{acc:.2f}").
  • dataset_manager/predefined/bfcl_v4/__init__.py:180generate() selection logic untested. It defines the frozen ~995-sample MLPerf gate composition. Untested: unknown-category ValueError; multi-turn subset filtering + all-filtered empty-DataFrame warning; cached-parquet SHA re-verification; 0 < pct <= 100 validation; subset_floor full-take vs max(1, int(total*pct/100)); _resolve_subset_pct precedence; max_samples truncation. Testable with a synthetic frame monkeypatched past the load step. A regression here silently changes what the accuracy gate measures.

Deferred out of PR #346.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.