[Bug]: LocalEvalSampler treats explicit empty eval case ID lists as all cases

Open
#7,168 3 comments 0 reactions 1 assignee View on GitHub

@surajksharma07 is already working on this.

Since Sep 18, 2026.

Assessment

This issue has not been assessed yet.

Description

eval request clarification

🔴 Required Information

Describe the Bug:

LocalEvalSamplerConfig documents train_eval_case_ids and validation_eval_case_ids as optional lists where all cases are selected when a field is not provided. However, LocalEvalSampler.__init__ uses truthiness checks, so explicitly passing an empty list is indistinguishable from passing None.

As a result, train_eval_case_ids=[] expands to every training case, and validation_eval_case_ids=[] expands to every validation case (or inherits the training cases). This can unexpectedly turn a no-case selection into a full evaluation run.

Steps to Reproduce:

  1. Install the current google-adk repository in an isolated environment.
  2. Construct LocalEvalSamplerConfig with explicit empty training and validation case ID lists.
  3. Construct LocalEvalSampler while returning known IDs from _get_eval_case_ids.
  4. Inspect get_train_example_ids() and get_validation_example_ids().

Minimal code is included below.

Expected Behavior:

Only None / an omitted field should trigger the documented “all eval cases” fallback. An explicit empty list should either remain empty, consistent with sample_and_score(batch=[]), or be rejected with a clear validation error if empty datasets are unsupported. It should not silently expand to all cases.

Observed Behavior:

The explicit empty lists are replaced with every ID returned for their respective eval sets:

configured train: []
actual train: ['train-1', 'train-2']
configured validation: []
actual validation: ['validation-1']
lookup calls: [('train',), ('validation',)]

Environment Details:

  • ADK Library Version: editable install from commit 3f24d2036a3434b755f6337026ac19a737041f85 (google-adk 2.9.0)
  • Desktop OS: Windows 10.0.26200
  • Python Version: Python 3.12 (64-bit)

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: N/A — the issue occurs during sampler construction before any model call

🟡 Optional Information

Regression:

Unknown. The truthiness behavior is present in the commit that originally introduced LocalEvalSampler.

Logs:

N/A — no external service or model call is required.

Screenshots / Video:

N/A.

Additional Context:

The constructor currently uses:

self._config.train_eval_case_ids or self._get_eval_case_ids(...)

and:

if self._config.validation_eval_case_ids:

Both checks collapse [] and None even though the Pydantic model and field descriptions distinguish them. The existing parameterized constructor test covers omitted and non-empty lists, but not explicit empty lists.

I searched the repository issues and pull requests for train_eval_case_ids and validation_eval_case_ids and found no existing report or implementation. I would be happy to contribute a focused fix and regression tests once the intended empty-list behavior is confirmed.

Minimal Reproduction Code:

from unittest.mock import MagicMock, patch

from google.adk.evaluation.eval_config import EvalConfig
from google.adk.evaluation.eval_sets_manager import EvalSetsManager
from google.adk.optimization.local_eval_sampler import LocalEvalSampler
from google.adk.optimization.local_eval_sampler import LocalEvalSamplerConfig

config = LocalEvalSamplerConfig(
    eval_config=EvalConfig(),
    app_name="app",
    train_eval_set="train",
    train_eval_case_ids=[],
    validation_eval_set="validation",
    validation_eval_case_ids=[],
)

with patch.object(
    LocalEvalSampler,
    "_get_eval_case_ids",
    side_effect=lambda eval_set_id: (
        ["train-1", "train-2"]
        if eval_set_id == "train"
        else ["validation-1"]
    ),
):
  sampler = LocalEvalSampler(config, MagicMock(spec=EvalSetsManager))

print("configured train:", config.train_eval_case_ids)
print("actual train:", sampler.get_train_example_ids())
print("configured validation:", config.validation_eval_case_ids)
print("actual validation:", sampler.get_validation_example_ids())

How often has this issue occurred?:

  • Always (100%)
Dominant language
Python
Stars
21.6k
Forks
4k
Avg merge
13h 49m
Merged PRs (30d)
10

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from google/adk-python

All issues in google/adk-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.