mlcommons / mlcommons/endpoints

BFCL multi-turn execution robustness: malformed 200 body + uncleaned bfcl-eval instance cache

Open
#392 0 comments 0 reactions 1 assignee View on GitHub

@Palanivelg is already working on this.

Since Jul 22, 2026.

Dominant language
Python
Stars
21
Forks
28
Avg merge
3d 17h
Merged PRs (30d)
13

Description

From PR #346 review (2026-07-02).

1. Malformed 200 body silently degrades accuracy (bfcl_v4_multi_turn_runner.py:236).
A structurally malformed 200 body (proxy error page, {"error": ...} payload, empty choices) returns (None, None, None), which process_response treats as "no tool calls" (bfcl_v4_execution.py:192-199) — it appends a fabricated empty assistant message and advances the turn, scoring the entry as a normal completion with degraded accuracy. This is inconsistent with the transport-error path (_send_request returning None force-terminates). A malformed body is the same failure class as an HTTP 500 and should force-terminate (or be surfaced distinctly); otherwise a flaky gateway silently degrades reported accuracy.

2. cleanup() does not free bfcl-eval's cached class instances (bfcl_v4_execution.py:316).
The simulated class instances bfcl-eval creates are cached in its module globals() keyed f"{model_name}_{test_entry_id}_{class_name}_instance" and never freed. Consequences: (a) re-running an entry in one process resumes from the first run's mutated state instead of initial_config (wrong results, no error); (b) a full run accumulates thousands of instances for the process lifetime. Since _MODEL_NAME_FOR_EXECUTION is fixed, the cache key is predictable — cleanup() can delete the matching globals() entries.

Deferred out of PR #346.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.