[Bug]: LocalEvalSampler treats explicit empty eval case ID lists as all cases
@surajksharma07 ya está trabajando en esto.
Desde el 18/9/2026.
Evaluación
Este issue todavía no se ha evaluado.
Descripción
🔴 Required Information
Describe the Bug:
LocalEvalSamplerConfig documents train_eval_case_ids and validation_eval_case_ids as optional lists where all cases are selected when a field is not provided. However, LocalEvalSampler.__init__ uses truthiness checks, so explicitly passing an empty list is indistinguishable from passing None.
As a result, train_eval_case_ids=[] expands to every training case, and validation_eval_case_ids=[] expands to every validation case (or inherits the training cases). This can unexpectedly turn a no-case selection into a full evaluation run.
Steps to Reproduce:
- Install the current
google-adkrepository in an isolated environment. - Construct
LocalEvalSamplerConfigwith explicit empty training and validation case ID lists. - Construct
LocalEvalSamplerwhile returning known IDs from_get_eval_case_ids. - Inspect
get_train_example_ids()andget_validation_example_ids().
Minimal code is included below.
Expected Behavior:
Only None / an omitted field should trigger the documented “all eval cases” fallback. An explicit empty list should either remain empty, consistent with sample_and_score(batch=[]), or be rejected with a clear validation error if empty datasets are unsupported. It should not silently expand to all cases.
Observed Behavior:
The explicit empty lists are replaced with every ID returned for their respective eval sets:
configured train: []
actual train: ['train-1', 'train-2']
configured validation: []
actual validation: ['validation-1']
lookup calls: [('train',), ('validation',)]
Environment Details:
- ADK Library Version: editable install from commit
3f24d2036a3434b755f6337026ac19a737041f85(google-adk 2.9.0) - Desktop OS: Windows 10.0.26200
- Python Version: Python 3.12 (64-bit)
Model Information:
- Are you using LiteLLM: No
- Which model is being used: N/A — the issue occurs during sampler construction before any model call
🟡 Optional Information
Regression:
Unknown. The truthiness behavior is present in the commit that originally introduced LocalEvalSampler.
Logs:
N/A — no external service or model call is required.
Screenshots / Video:
N/A.
Additional Context:
The constructor currently uses:
self._config.train_eval_case_ids or self._get_eval_case_ids(...)
and:
if self._config.validation_eval_case_ids:
Both checks collapse [] and None even though the Pydantic model and field descriptions distinguish them. The existing parameterized constructor test covers omitted and non-empty lists, but not explicit empty lists.
I searched the repository issues and pull requests for train_eval_case_ids and validation_eval_case_ids and found no existing report or implementation. I would be happy to contribute a focused fix and regression tests once the intended empty-list behavior is confirmed.
Minimal Reproduction Code:
from unittest.mock import MagicMock, patch
from google.adk.evaluation.eval_config import EvalConfig
from google.adk.evaluation.eval_sets_manager import EvalSetsManager
from google.adk.optimization.local_eval_sampler import LocalEvalSampler
from google.adk.optimization.local_eval_sampler import LocalEvalSamplerConfig
config = LocalEvalSamplerConfig(
eval_config=EvalConfig(),
app_name="app",
train_eval_set="train",
train_eval_case_ids=[],
validation_eval_set="validation",
validation_eval_case_ids=[],
)
with patch.object(
LocalEvalSampler,
"_get_eval_case_ids",
side_effect=lambda eval_set_id: (
["train-1", "train-2"]
if eval_set_id == "train"
else ["validation-1"]
),
):
sampler = LocalEvalSampler(config, MagicMock(spec=EvalSetsManager))
print("configured train:", config.train_eval_case_ids)
print("actual train:", sampler.get_train_example_ids())
print("configured validation:", config.validation_eval_case_ids)
print("actual validation:", sampler.get_validation_example_ids())
How often has this issue occurred?:
- Always (100%)
- Lenguaje dominante
- Python
- Estrellas
- 21.6k
- Forks
- 4k
- Merge medio
- 13 h 49 min
- PR fusionados (30 d)
- 10
Guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de google/adk-python
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
google/adk-python#7217 · 1 comentario ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
google/adk-python#7206 · 1 comentario ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 86/100
google/adk-python#7205 · 1 comentario ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
google/adk-python#7196 · 1 comentario ·
-
eval request clarification
Dificultad 1/5 1-3 horas Aptitud para principiantes 86/100
google/adk-python#7146 · 2 comentarios · 1 asignado ·
Todos los issues de google/adk-python
Issues similares
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 86/100
zostera/django-bootstrap4#894 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
use-agent-os/agent-os#3276 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 88/100
zephyrproject-rtos/zephyr#119726 ·
-
area/auth bug comp/agent P3 platform/discord type/security
Dificultad 2/5 1-3 horas Aptitud para principiantes 88/100
NousResearch/hermes-agent#117848 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
zilliztech/memsearch#759 ·