mlcommons / mlcommons/endpoints
BFCL test coverage: BFCLv4MultiTurnScorer.score() and bfcl_v4 generate() selection logic
Open
@Palanivelg is already working on this.
Since Jul 22, 2026.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
From PR #346 review (2026-07-02).
bfcl_v4_multi_turn_scorer.py:59—score()untested. Untested branches:force_terminatedearly-fail,turn_count_mismatch, thechecker_exceptioncatch, valid/invalid bookkeeping, unweighted-mean category aggregation, and the empty-valid_subset_accs-> 0.0 branch. Can be tested by patchingmulti_turn_checker(same approach as the runner/execution-bridge tests). Also pin the string-typedsubset_scorescontract (f"{acc:.2f}").dataset_manager/predefined/bfcl_v4/__init__.py:180—generate()selection logic untested. It defines the frozen ~995-sample MLPerf gate composition. Untested: unknown-categoryValueError; multi-turn subset filtering + all-filtered empty-DataFrame warning; cached-parquet SHA re-verification;0 < pct <= 100validation;subset_floorfull-take vsmax(1, int(total*pct/100));_resolve_subset_pctprecedence;max_samplestruncation. Testable with a synthetic frame monkeypatched past the load step. A regression here silently changes what the accuracy gate measures.
Deferred out of PR #346.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.