[Bug]: LocalEvalSampler treats explicit empty eval case ID lists as all cases
@surajksharma07 is already working on this.
Since Sep 18, 2026.
Assessment
This issue has not been assessed yet.
Description
🔴 Required Information
Describe the Bug:
LocalEvalSamplerConfig documents train_eval_case_ids and validation_eval_case_ids as optional lists where all cases are selected when a field is not provided. However, LocalEvalSampler.__init__ uses truthiness checks, so explicitly passing an empty list is indistinguishable from passing None.
As a result, train_eval_case_ids=[] expands to every training case, and validation_eval_case_ids=[] expands to every validation case (or inherits the training cases). This can unexpectedly turn a no-case selection into a full evaluation run.
Steps to Reproduce:
- Install the current
google-adkrepository in an isolated environment. - Construct
LocalEvalSamplerConfigwith explicit empty training and validation case ID lists. - Construct
LocalEvalSamplerwhile returning known IDs from_get_eval_case_ids. - Inspect
get_train_example_ids()andget_validation_example_ids().
Minimal code is included below.
Expected Behavior:
Only None / an omitted field should trigger the documented “all eval cases” fallback. An explicit empty list should either remain empty, consistent with sample_and_score(batch=[]), or be rejected with a clear validation error if empty datasets are unsupported. It should not silently expand to all cases.
Observed Behavior:
The explicit empty lists are replaced with every ID returned for their respective eval sets:
configured train: []
actual train: ['train-1', 'train-2']
configured validation: []
actual validation: ['validation-1']
lookup calls: [('train',), ('validation',)]
Environment Details:
- ADK Library Version: editable install from commit
3f24d2036a3434b755f6337026ac19a737041f85(google-adk 2.9.0) - Desktop OS: Windows 10.0.26200
- Python Version: Python 3.12 (64-bit)
Model Information:
- Are you using LiteLLM: No
- Which model is being used: N/A — the issue occurs during sampler construction before any model call
🟡 Optional Information
Regression:
Unknown. The truthiness behavior is present in the commit that originally introduced LocalEvalSampler.
Logs:
N/A — no external service or model call is required.
Screenshots / Video:
N/A.
Additional Context:
The constructor currently uses:
self._config.train_eval_case_ids or self._get_eval_case_ids(...)
and:
if self._config.validation_eval_case_ids:
Both checks collapse [] and None even though the Pydantic model and field descriptions distinguish them. The existing parameterized constructor test covers omitted and non-empty lists, but not explicit empty lists.
I searched the repository issues and pull requests for train_eval_case_ids and validation_eval_case_ids and found no existing report or implementation. I would be happy to contribute a focused fix and regression tests once the intended empty-list behavior is confirmed.
Minimal Reproduction Code:
from unittest.mock import MagicMock, patch
from google.adk.evaluation.eval_config import EvalConfig
from google.adk.evaluation.eval_sets_manager import EvalSetsManager
from google.adk.optimization.local_eval_sampler import LocalEvalSampler
from google.adk.optimization.local_eval_sampler import LocalEvalSamplerConfig
config = LocalEvalSamplerConfig(
eval_config=EvalConfig(),
app_name="app",
train_eval_set="train",
train_eval_case_ids=[],
validation_eval_set="validation",
validation_eval_case_ids=[],
)
with patch.object(
LocalEvalSampler,
"_get_eval_case_ids",
side_effect=lambda eval_set_id: (
["train-1", "train-2"]
if eval_set_id == "train"
else ["validation-1"]
),
):
sampler = LocalEvalSampler(config, MagicMock(spec=EvalSetsManager))
print("configured train:", config.train_eval_case_ids)
print("actual train:", sampler.get_train_example_ids())
print("configured validation:", config.validation_eval_case_ids)
print("actual validation:", sampler.get_validation_example_ids())
How often has this issue occurred?:
- Always (100%)
- Dominant language
- Python
- Stars
- 21.6k
- Forks
- 4k
- Avg merge
- 13h 49m
- Merged PRs (30d)
- 10
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from google/adk-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
google/adk-python#7217 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
google/adk-python#7206 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
google/adk-python#7205 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
google/adk-python#7196 · 1 comment ·
-
eval request clarification
Difficulty 1/5 1-3 hours Newbie friendliness 86/100
google/adk-python#7146 · 2 comments · 1 assignee ·
All issues in google/adk-python
Similar issues
-
area/auth bug comp/agent P3 platform/discord type/security
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
NousResearch/hermes-agent#117848 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100