google / google/adk-python

evaluate_eval_set silently passes when eval_results_by_eval_id is empty (zero eval cases or num_runs=0)

Cerrado
#6,951 6 comentarios 0 reacciones 1 asignado Reclamado por @surajksharma07 Ver en GitHub
eval
Lenguaje dominante
Python
Estrellas
21.5k
Forks
4k
Merge medio
1 d 14 h
PR fusionados (30 d)
37

Descripción

## 🔴 Required Information

**Describe the Bug:**
`AgentEvaluator.evaluate_eval_set` reports overall success on an eval set that
was never actually evaluated. `evaluate_eval_set` builds `eval_results_by_eval_id`
from the eval set, iterates it to populate a `failures` list, and finishes with
`assert not failures`. If `eval_results_by_eval_id` ends up empty for any
reason, the loop body never runs, `failures` stays `[]`, and the assert passes
— the function returns normally with no signal that zero cases were evaluated.

Two independent, ordinary public-API calls reach this with no validation
bypass:
1. An `EvalSet` with `eval_cases=[]` — `EvalSet.eval_cases` has no
`min_length`/validator, so this constructs cleanly.
2. `evaluate_eval_set(..., num_runs=0)` on a *non-empty* eval set — `num_runs`
has no bounds check anywhere in the file, and `inference_requests =
[InferenceRequest(...)] * num_runs` becomes `[]`.

Both silently produce the same "nothing was evaluated, but nothing failed
either" outcome.

**Steps to Reproduce:** minimal, self-contained repro (writes a real temp
agent module, no third-party packages):

```python
"""Minimal, self-contained repro for AgentEvaluator.evaluate_eval_set's
vacuous pass on an empty eval set (agent_evaluator.py, evaluate_eval_set).
"""
from __future__ import annotations
import asyncio, tempfile, uuid
from pathlib import Path
from google.adk.evaluation.agent_evaluator import AgentEvaluator
from google.adk.evaluation.eval_config import EvalConfig
from google.adk.evaluation.eval_set import EvalSet

def _write_agent_module(tmp_path: Path) -> str:
package_name = f"vacuous_pass_repro_{uuid.uuid4().hex}"
package_dir = tmp_path / package_name
package_dir.mkdir()
(package_dir / "__init__.py").write_text("", encoding="utf-8")
(package_dir / "agent.py").write_text(
"from google.adk.agents.llm_agent import LlmAgent\n"
"from google.adk.models.base_llm import BaseLlm\n"
"from google.adk.models.llm_response import LlmResponse\n"
"from google.genai import types as genai_types\n\n\n"
"class _FakeLlm(BaseLlm):\n"
" model: str = 'fake-model'\n\n"
" @classmethod\n"
" def supported_models(cls):\n"
" return ['fake-model']\n\n"
" async def generate_content_async(self, llm_request, stream=False):\n"
" yield LlmResponse(content=genai_types.Content(\n"
" parts=[genai_types.Part(text='ok')], role='model'))\n\n\n"
"root_agent = LlmAgent(name='vacuous_pass_repro_agent', model=_FakeLlm(),\n"
" instruction='Answer briefly.')\n",
encoding="utf-8",
)
return f"{package_name}.agent"

async def main() -> None:
with tempfile.TemporaryDirectory() as tmp_dir_str:
tmp_path = Path(tmp_dir_str)
module_name = _write_agent_module(tmp_path)
import sys
sys.path.insert(0, str(tmp_path))
try:
empty_eval_set = EvalSet(eval_set_id="empty_set", eval_cases=[])
result = await AgentEvaluator.evaluate_eval_set(
agent_module=module_name,
eval_set=empty_eval_set,
eval_config=EvalConfig(criteria={"tool_trajectory_avg_score": 1.0}),
)
print(f"RESULT: completed WITHOUT raising. Returned: {result!r}")
print("VERDICT: PASSED -- vacuous pass on zero eval cases.")
finally:
sys.path.remove(str(tmp_path))
sys.modules.pop(module_name, None)
sys.modules.pop(module_name.rsplit(".", 1)[0], None)

if __name__ == "__main__":
asyncio.run(main())
```

**Expected Behavior:** `evaluate_eval_set` should raise or otherwise signal
that zero cases were evaluated — a "pass" should mean "every requested case
was checked and none failed," not "nothing was checked."

**Observed Behavior:**
```
RESULT: completed WITHOUT raising. Returned: None
VERDICT: PASSED -- vacuous pass on zero eval cases.
```
Process exit code `0`. A second, independent repro (non-empty `EvalSet`,
`num_runs=0`) reproduces the identical outcome through the same mechanism.

**Environment:** google-adk main branch (this repo),
`src/google/adk/evaluation/agent_evaluator.py:289` (the `assert not failures`
line), `AgentEvaluator.evaluate_eval_set`.

## CI implication

Any test suite or CI gate that calls `AgentEvaluator.evaluate_eval_set`/`.evaluate()`
directly (rather than going through the `adk eval` CLI) will report a clean
pass for an eval set that was silently emptied by a config bug, a bad filter,
or an accidental `num_runs=0`, rather than failing loudly.

**Note:** this is a separate code path from the `adk eval` CLI command
(`cli_tools_click.py::cli_eval`, which builds its own `eval_run_summary` via
`LocalEvalService` directly and never calls `AgentEvaluator`) — confirmed by
reading both call chains. The CLI has an analogous "empty collection defaults
to a pass" shape in its own exit-code logic, which is unaffected by this
issue's scope either way.

I'm happy to send a PR for this if useful — the fix is a small, additive
check (e.g. treat an empty `eval_results_by_eval_id` as `NOT_EVALUATED`
rather than a silent pass), similar in shape to a couple of other
NOT_EVALUATED-masking fixes already proposed against this same subsystem.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.