[Bug]: LocalEvalSampler treats explicit empty eval case ID lists as all cases

オープン
#7,168 コメント 3 件 リアクション 0 件 担当者 1 名 GitHub で見る

@surajksharma07 がすでに取り組んでいます。

2026年9月18日 から。

評価

この issue はまだ評価されていません。

説明

eval request clarification

🔴 Required Information

Describe the Bug:

LocalEvalSamplerConfig documents train_eval_case_ids and validation_eval_case_ids as optional lists where all cases are selected when a field is not provided. However, LocalEvalSampler.__init__ uses truthiness checks, so explicitly passing an empty list is indistinguishable from passing None.

As a result, train_eval_case_ids=[] expands to every training case, and validation_eval_case_ids=[] expands to every validation case (or inherits the training cases). This can unexpectedly turn a no-case selection into a full evaluation run.

Steps to Reproduce:

  1. Install the current google-adk repository in an isolated environment.
  2. Construct LocalEvalSamplerConfig with explicit empty training and validation case ID lists.
  3. Construct LocalEvalSampler while returning known IDs from _get_eval_case_ids.
  4. Inspect get_train_example_ids() and get_validation_example_ids().

Minimal code is included below.

Expected Behavior:

Only None / an omitted field should trigger the documented “all eval cases” fallback. An explicit empty list should either remain empty, consistent with sample_and_score(batch=[]), or be rejected with a clear validation error if empty datasets are unsupported. It should not silently expand to all cases.

Observed Behavior:

The explicit empty lists are replaced with every ID returned for their respective eval sets:

configured train: []
actual train: ['train-1', 'train-2']
configured validation: []
actual validation: ['validation-1']
lookup calls: [('train',), ('validation',)]

Environment Details:

  • ADK Library Version: editable install from commit 3f24d2036a3434b755f6337026ac19a737041f85 (google-adk 2.9.0)
  • Desktop OS: Windows 10.0.26200
  • Python Version: Python 3.12 (64-bit)

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: N/A — the issue occurs during sampler construction before any model call

🟡 Optional Information

Regression:

Unknown. The truthiness behavior is present in the commit that originally introduced LocalEvalSampler.

Logs:

N/A — no external service or model call is required.

Screenshots / Video:

N/A.

Additional Context:

The constructor currently uses:

self._config.train_eval_case_ids or self._get_eval_case_ids(...)

and:

if self._config.validation_eval_case_ids:

Both checks collapse [] and None even though the Pydantic model and field descriptions distinguish them. The existing parameterized constructor test covers omitted and non-empty lists, but not explicit empty lists.

I searched the repository issues and pull requests for train_eval_case_ids and validation_eval_case_ids and found no existing report or implementation. I would be happy to contribute a focused fix and regression tests once the intended empty-list behavior is confirmed.

Minimal Reproduction Code:

from unittest.mock import MagicMock, patch

from google.adk.evaluation.eval_config import EvalConfig
from google.adk.evaluation.eval_sets_manager import EvalSetsManager
from google.adk.optimization.local_eval_sampler import LocalEvalSampler
from google.adk.optimization.local_eval_sampler import LocalEvalSamplerConfig

config = LocalEvalSamplerConfig(
    eval_config=EvalConfig(),
    app_name="app",
    train_eval_set="train",
    train_eval_case_ids=[],
    validation_eval_set="validation",
    validation_eval_case_ids=[],
)

with patch.object(
    LocalEvalSampler,
    "_get_eval_case_ids",
    side_effect=lambda eval_set_id: (
        ["train-1", "train-2"]
        if eval_set_id == "train"
        else ["validation-1"]
    ),
):
  sampler = LocalEvalSampler(config, MagicMock(spec=EvalSetsManager))

print("configured train:", config.train_eval_case_ids)
print("actual train:", sampler.get_train_example_ids())
print("configured validation:", config.validation_eval_case_ids)
print("actual validation:", sampler.get_validation_example_ids())

How often has this issue occurred?:

  • Always (100%)
主要言語
Python
スター
21.6k
フォーク
4k
平均マージ
13時間 49分
マージ済み PR(30日)
10

コントリビューションガイド

コントリビューションガイドを開く

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

google/adk-python のほかの issue

google/adk-python の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。