[Bug]: LocalEvalSampler treats explicit empty eval case ID lists as all cases
@surajksharma07 がすでに取り組んでいます。
2026年9月18日 から。
評価
この issue はまだ評価されていません。
説明
🔴 Required Information
Describe the Bug:
LocalEvalSamplerConfig documents train_eval_case_ids and validation_eval_case_ids as optional lists where all cases are selected when a field is not provided. However, LocalEvalSampler.__init__ uses truthiness checks, so explicitly passing an empty list is indistinguishable from passing None.
As a result, train_eval_case_ids=[] expands to every training case, and validation_eval_case_ids=[] expands to every validation case (or inherits the training cases). This can unexpectedly turn a no-case selection into a full evaluation run.
Steps to Reproduce:
- Install the current
google-adkrepository in an isolated environment. - Construct
LocalEvalSamplerConfigwith explicit empty training and validation case ID lists. - Construct
LocalEvalSamplerwhile returning known IDs from_get_eval_case_ids. - Inspect
get_train_example_ids()andget_validation_example_ids().
Minimal code is included below.
Expected Behavior:
Only None / an omitted field should trigger the documented “all eval cases” fallback. An explicit empty list should either remain empty, consistent with sample_and_score(batch=[]), or be rejected with a clear validation error if empty datasets are unsupported. It should not silently expand to all cases.
Observed Behavior:
The explicit empty lists are replaced with every ID returned for their respective eval sets:
configured train: []
actual train: ['train-1', 'train-2']
configured validation: []
actual validation: ['validation-1']
lookup calls: [('train',), ('validation',)]
Environment Details:
- ADK Library Version: editable install from commit
3f24d2036a3434b755f6337026ac19a737041f85(google-adk 2.9.0) - Desktop OS: Windows 10.0.26200
- Python Version: Python 3.12 (64-bit)
Model Information:
- Are you using LiteLLM: No
- Which model is being used: N/A — the issue occurs during sampler construction before any model call
🟡 Optional Information
Regression:
Unknown. The truthiness behavior is present in the commit that originally introduced LocalEvalSampler.
Logs:
N/A — no external service or model call is required.
Screenshots / Video:
N/A.
Additional Context:
The constructor currently uses:
self._config.train_eval_case_ids or self._get_eval_case_ids(...)
and:
if self._config.validation_eval_case_ids:
Both checks collapse [] and None even though the Pydantic model and field descriptions distinguish them. The existing parameterized constructor test covers omitted and non-empty lists, but not explicit empty lists.
I searched the repository issues and pull requests for train_eval_case_ids and validation_eval_case_ids and found no existing report or implementation. I would be happy to contribute a focused fix and regression tests once the intended empty-list behavior is confirmed.
Minimal Reproduction Code:
from unittest.mock import MagicMock, patch
from google.adk.evaluation.eval_config import EvalConfig
from google.adk.evaluation.eval_sets_manager import EvalSetsManager
from google.adk.optimization.local_eval_sampler import LocalEvalSampler
from google.adk.optimization.local_eval_sampler import LocalEvalSamplerConfig
config = LocalEvalSamplerConfig(
eval_config=EvalConfig(),
app_name="app",
train_eval_set="train",
train_eval_case_ids=[],
validation_eval_set="validation",
validation_eval_case_ids=[],
)
with patch.object(
LocalEvalSampler,
"_get_eval_case_ids",
side_effect=lambda eval_set_id: (
["train-1", "train-2"]
if eval_set_id == "train"
else ["validation-1"]
),
):
sampler = LocalEvalSampler(config, MagicMock(spec=EvalSetsManager))
print("configured train:", config.train_eval_case_ids)
print("actual train:", sampler.get_train_example_ids())
print("configured validation:", config.validation_eval_case_ids)
print("actual validation:", sampler.get_validation_example_ids())
How often has this issue occurred?:
- Always (100%)
- 主要言語
- Python
- スター
- 21.6k
- フォーク
- 4k
- 平均マージ
- 13時間 49分
- マージ済み PR(30日)
- 10
コントリビューションガイド
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
google/adk-python のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
google/adk-python#7217 · コメント 1 件 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
google/adk-python#7206 · コメント 1 件 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 86/100
google/adk-python#7205 · コメント 1 件 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
google/adk-python#7196 · コメント 1 件 ·
-
eval request clarification
難易度 1/5 1〜3時間 初心者へのやさしさ 86/100
google/adk-python#7146 · コメント 2 件 · 担当者 1 名 ·
google/adk-python の issue をすべて見る
似ている issue
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 86/100
zostera/django-bootstrap4#894 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
use-agent-os/agent-os#3276 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 88/100
zephyrproject-rtos/zephyr#119726 ·
-
area/auth bug comp/agent P3 platform/discord type/security
難易度 2/5 1〜3時間 初心者へのやさしさ 88/100
NousResearch/hermes-agent#117848 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
zilliztech/memsearch#759 ·