ExperimentRunner._assert_scores returns True when expected_score is empty — case recorded as passed with zero criteria evaluated

Description

Note: this is a source-review finding, not a runtime report. I have not run crewAI, so the Operating System / Python Version / Virtual Environment fields below are template defaults and are not meaningful here — the defect is environment-independent and visible in the source. Happy to move this to a discussion if a static finding isn't wanted as a bug report.

ExperimentRunner._assert_scores() returns True unconditionally when a dataset entry's expected_score is an empty dict, so the case is recorded passed=True with zero criteria evaluated.

In lib/crewai/src/crewai/experimental/evaluation/experiment/runner.py:

• when expected is a dict and actual is a scalar, the verdict is all(actual >= exp_score for exp_score in expected.values()). all() over an empty generator is True.
• when both are dicts, there is an explicit if not expected: return True.

Either way the result flows into ExperimentResultsDisplay's summary and into compare_with_baseline's improved/regressed labels as a pass.

Severity is low — the dataset is authored by the same developer running the eval, nothing in the library produces an empty expected_score by default, and it is not reachable by an untrusted input. But an eval harness reporting a pass over an empty check-set is the failure it exists to prevent: a templating bug or an unfilled placeholder in a dataset entry silently converts a real regression into a green run.

Steps to Reproduce

Observed by reading the source at main commit c3f83cd, not by executing it. The code path is unambiguous, and a runtime reproduction would be:

1. Build an experiment dataset in which one entry's expected_score is an empty dict:

   from crewai.experimental.evaluation.experiment.runner import ExperimentRunner
   ExperimentRunner(dataset=[{"inputs": {...}, "expected_score": {}}]).run(agents=[...])
   

2. Run it against an agent whose actual score is arbitrarily bad.
3. Inspect the results summary.
4. The case is reported as passed, and is counted as a pass by compare_with_baseline, although no criterion was compared.

The same holds whether the agent's actual score is a scalar (hits the all(...) branch) or a dict (hits the explicit if not expected: return True).

Expected behavior

An empty expected_score should not produce a pass. A case for which no criterion was compared is not a case that passed.

Either of these would be reasonable:

• treat an empty expected_score as SKIPPED / INCONCLUSIVE and exclude it from the pass/fail tallies, or
• raise on it, on the grounds that an empty expectation is almost certainly a mistake in the dataset rather than an intentional "no assertions" case.

What should not happen is the current behaviour, where the run is reported green and compare_with_baseline counts it as an improvement or a non-regression.

Screenshots/Code snippets

lib/crewai/src/crewai/experimental/evaluation/experiment/runner.py, in _assert_scores, at main commit c3f83cd:

if isinstance(expected, dict) and isinstance(actual, (int, float)):
    return all(actual >= exp_score for exp_score in expected.values())
    # expected == {}  ->  all(<empty>)  ->  True

if isinstance(expected, (int, float)) and isinstance(actual, dict):
    if not actual:
        return False          # <- the mirrored case IS guarded

if isinstance(expected, dict) and isinstance(actual, dict):
    if not expected:
        return True           # <- empty expectation passes explicitly
    matching_keys = set(expected.keys()) & set(actual.keys())
    if not matching_keys:
        return False          # <- and this one is guarded too

Worth noting the asymmetry, which is what suggests the empty-expected behaviour is an oversight rather than a decision: an empty actual returns False, and an empty key intersection returns False, but an empty expected returns True.

Operating System

Ubuntu 20.04

Python Version

3.10

crewAI Version

main @ c3f83cd (2026-09-17) — identified by source review; no installed version

crewAI Tools Version

n/a — not installed; source review only

Virtual Environment

Venv

Evidence

No logs, because this was not observed at runtime — the evidence is the source itself, quoted in full under "Screenshots/Code snippets" above.

The three call sites that consume the verdict, and are therefore what a false pass propagates into:

• _assert_scores returns into the per-case result that carries passed
• ExperimentResultsDisplay tallies those into the run summary
• compare_with_baseline uses them for the improved / regressed / unchanged labels

So a case with an empty expected_score is counted as a pass in the summary and as a non-regression against a baseline, with no indication that nothing was checked.

Possible Solution

Make the empty expectation an explicit outcome rather than a pass. Minimal version, matching the guards already present for the mirrored cases:

if isinstance(expected, dict) and not expected:
    raise ValueError(
        "expected_score is an empty dict; no criteria to assert against. "
        "Provide at least one metric threshold, or omit the case."
    )

placed before the type dispatch, so both the all(...) branch and the if not expected: return True branch become unreachable for the empty case.

If raising is too strict — say some workflows legitimately carry placeholder cases — the alternative is a third state alongside pass/fail (SKIPPED), excluded from the summary tallies and from compare_with_baseline, so an unchecked case can never read as a green one.

Either is fine by me; the current behaviour is the only one I'd argue against. Happy to open a PR once you've picked.

Additional context

Found by a static search for verification functions whose verdict is true on empty input — all([]), not any([]), if not items: return True, len(...) == 0 — run across several open-source agent frameworks. 41 candidates surfaced across four of them; all 41 were adjudicated by hand against three criteria (the verdict must gate something, the empty case must be reachable, and empty must mean nothing was examined rather than nothing needed examining). Most were false positives of the heuristic itself — all() over a fixed non-empty literal, or a vacuous True that triggers a raise and so fails closed. This was one of only two that survived adjudication, and the only one in crewAI.

Happy to open a PR if the maintainers have a preference between skipping and raising.

Start in lib/crewai/src/crewai/experimental/evaluation/experiment/runner.py at _assert_scores and trace its per-case result into ExperimentResultsDisplay and compare_with_baseline. Reproduce an entry with an empty expected_score using the dataset example, then verify that the chosen outcome is not counted as a pass or non-regression and that normal scalar and dictionary expectations remain unchanged.