mlcommons / mlcommons/endpoints

Move bfcl-eval to an out-of-process uv subproject (remove parent-env pip caveats)

Open
#390 0 comments 0 reactions 1 assignee View on GitHub

@Palanivelg is already working on this.

Since Jul 22, 2026.

Dominant language
Python
Stars
21
Forks
28
Avg merge
3d 17h
Merged PRs (30d)
13

Description

Context

Raised in PR #346 review (pyproject.toml:78).

bfcl-eval is currently a parent-env optional dependency (the [bfcl] extra). uv-side isolation ([tool.uv].conflicts + the filelock/virtualenv CVE floors) is airtight for uv sync, but it leaks for pip, which ignores both uv.lock and conflicts:

  • we had to relax numpy==2.4.4 -> numpy>=1.26.4 (pyproject.toml:78), so pip install -e . no longer pins numpy reproducibly;
  • pip install -e ".[bfcl,dev]" hard-fails on the filelock floor.

This is a regression from the repo's exact-pin convention, forced by needing bfcl-eval installable in the parent env.

Proposal

Move bfcl-eval out of the parent env into an out-of-process uv run --project subproject, mirroring the existing pattern for:

  • VBench (examples/09_Wan22_VideoGen_Example/accuracy/)
  • DeepSeek-R1 (src/inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/)

bfcl-eval is a strong fit: it's already lazy-imported everywhere (bfcl_v4_scorer.py:42, multi_turn.py:144), and single-turn loading is bfcl-eval-free on the R2 parquet path (pre-formatted messages+tools) - only scoring (ast_checker) pulls it in. Multi-turn needs bfcl-eval in-process for live stateful execution (bfcl_v4_execution.py:37), but it's already its own CLI (bfcl_v4_multi_turn_cli), a natural subproject entrypoint.

Payoff

Moving scoring to a subproject would let the parent:

  • restore numpy==2.4.4 (exact pin);
  • drop the [bfcl] extra, the [tool.uv].conflicts block, and the filelock/virtualenv CVE floors;
  • remove the pip reproducibility caveats without changing the accuracy gate.
Scope

Deferred out of PR #346 to keep it focused on landing the workload + gate.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.