mlcommons / mlcommons/endpoints
Move bfcl-eval to an out-of-process uv subproject (remove parent-env pip caveats)
@Palanivelg is already working on this.
Since Jul 22, 2026.
- Dominant language
- Python
- Stars
- 21
- Forks
- 28
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 13
Description
Context
Raised in PR #346 review (pyproject.toml:78).
bfcl-eval is currently a parent-env optional dependency (the [bfcl] extra). uv-side isolation ([tool.uv].conflicts + the filelock/virtualenv CVE floors) is airtight for uv sync, but it leaks for pip, which ignores both uv.lock and conflicts:
- we had to relax
numpy==2.4.4->numpy>=1.26.4(pyproject.toml:78), sopip install -e .no longer pins numpy reproducibly; pip install -e ".[bfcl,dev]"hard-fails on thefilelockfloor.
This is a regression from the repo's exact-pin convention, forced by needing bfcl-eval installable in the parent env.
Proposal
Move bfcl-eval out of the parent env into an out-of-process uv run --project subproject, mirroring the existing pattern for:
- VBench (
examples/09_Wan22_VideoGen_Example/accuracy/) - DeepSeek-R1 (
src/inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/)
bfcl-eval is a strong fit: it's already lazy-imported everywhere (bfcl_v4_scorer.py:42, multi_turn.py:144), and single-turn loading is bfcl-eval-free on the R2 parquet path (pre-formatted messages+tools) - only scoring (ast_checker) pulls it in. Multi-turn needs bfcl-eval in-process for live stateful execution (bfcl_v4_execution.py:37), but it's already its own CLI (bfcl_v4_multi_turn_cli), a natural subproject entrypoint.
Payoff
Moving scoring to a subproject would let the parent:
- restore
numpy==2.4.4(exact pin); - drop the
[bfcl]extra, the[tool.uv].conflictsblock, and the filelock/virtualenv CVE floors; - remove the
pipreproducibility caveats without changing the accuracy gate.
Scope
Deferred out of PR #346 to keep it focused on landing the workload + gate.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.