mlcommons / mlcommons/endpoints
BFCL multi-turn execution robustness: malformed 200 body + uncleaned bfcl-eval instance cache
@Palanivelg is already working on this.
Since Jul 22, 2026.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
From PR #346 review (2026-07-02).
1. Malformed 200 body silently degrades accuracy (bfcl_v4_multi_turn_runner.py:236).
A structurally malformed 200 body (proxy error page, {"error": ...} payload, empty choices) returns (None, None, None), which process_response treats as "no tool calls" (bfcl_v4_execution.py:192-199) — it appends a fabricated empty assistant message and advances the turn, scoring the entry as a normal completion with degraded accuracy. This is inconsistent with the transport-error path (_send_request returning None force-terminates). A malformed body is the same failure class as an HTTP 500 and should force-terminate (or be surfaced distinctly); otherwise a flaky gateway silently degrades reported accuracy.
2. cleanup() does not free bfcl-eval's cached class instances (bfcl_v4_execution.py:316).
The simulated class instances bfcl-eval creates are cached in its module globals() keyed f"{model_name}_{test_entry_id}_{class_name}_instance" and never freed. Consequences: (a) re-running an entry in one process resumes from the first run's mutated state instead of initial_config (wrong results, no error); (b) a full run accumulates thousands of instances for the process lifetime. Since _MODEL_NAME_FOR_EXECUTION is fixed, the cache key is predictable — cleanup() can delete the matching globals() entries.
Deferred out of PR #346.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.