google / google/adk-python

Evaluator interface lacks access to session state

Đang mở
#4,532 1 bình luận 1 reaction 2 người được giao Được @ankursharmas nhận Xem trên GitHub
eval needs review
Ngôn ngữ chính
Python
Star
21.5k
Fork
4k
Merge trung bình
1 ngày 14 giờ
Pull request đã merge (30 ngày)
37

Mô tả

## 🔴 Required Information

### Is your feature request related to a specific problem?

The `Evaluator.evaluate_invocations` interface does not receive session state, making it impossible to build custom evaluators that assess the state an agent produces. The method signature only accepts `actual_invocations`, `expected_invocations`, and `conversation_scenario`:

```python
# google.adk.evaluation.evaluator.Evaluator
def evaluate_invocations(
self,
actual_invocations: list[Invocation],
expected_invocations: Optional[list[Invocation]] = None,
conversation_scenario: Optional[ConversationScenario] = None,
) -> EvaluationResult:
```

The `Invocation` model itself contains `user_content`, `final_response`, `intermediate_data`, `rubrics`, and `app_details` — but no session state.

Meanwhile, the data model already anticipates session state evaluation:
- `EvalCase.final_session_state: Optional[SessionState]` exists with the docstring *"The expected final session state at the end of the conversation"*
- `EvalCaseResult.session_details: Optional[Session]` is populated by `LocalEvalService` with the full session (including `Session.state`)

The plumbing exists, but the evaluation pipeline never wires session state through to evaluators — meaning `EvalCase.final_session_state` is defined but unused.

### Describe the Solution You'd Like

Extend the `Evaluator.evaluate_invocations` interface to optionally receive the initial and final session state, so that custom evaluators can access them:

```python
def evaluate_invocations(
self,
actual_invocations: list[Invocation],
expected_invocations: Optional[list[Invocation]] = None,
conversation_scenario: Optional[ConversationScenario] = None,
# New parameters:
initial_session_state: Optional[SessionState] = None,
final_session_state: Optional[SessionState] = None,
) -> EvaluationResult:
```

Where:
- `initial_session_state` is the state the session started with (from `EvalCase.session_input.state`)
- `final_session_state` is the state after inference completed (from `Session.state` via `EvalCaseResult.session_details`)

`LocalEvalService._evaluate_metric` would forward these from data already available at evaluation time. This is backward-compatible since both new parameters default to `None`, and existing evaluators can simply ignore them.

With this change, developers could build custom `Evaluator` subclasses that validate session state — compare it against `EvalCase.final_session_state` (the expected golden state), check that specific keys were populated, verify structural constraints, or use an LLM-as-judge over the state contents.

### Impact on your work

We are building multi-agent systems where agents collaboratively produce structured outputs stored in session state (e.g., timelines, assignments). The conversation is the means, but session state is the actual output artifact.

Currently, ADK's evaluation metrics can only evaluate text responses and tool calls, which represent a fraction of the agent's real output. The structured data we need to evaluate lives in session state and is invisible to the evaluator pipeline.

We have a workaround, but first-class support would let us leverage ADK's metric aggregation, reporting, and eval infrastructure for session state evaluation instead of maintaining a parallel post-processing pipeline.

### Willingness to contribute

No

---

## 🟡 Recommended Information

### Describe Alternatives You've Considered

We currently bypass the `Evaluator` interface entirely and run session-state-aware evaluation as a **post-processing step** after ADK's pipeline completes. We built a custom LLM judge class that receives `EvalCaseResult`, extracts conversation history from invocations, and additionally pulls structured data from `session_details.state` to include as context for the judge LLM.

**Why this is suboptimal:**
- It runs *outside* the ADK evaluation pipeline, so results aren't integrated into ADK's per-invocation metric tracking, aggregation, or `EvalMetricResult` reporting.
- It requires custom plumbing to collect `EvalSetResult` objects after `cli_eval` completes, run judges separately, and merge results into our own reporting format.
- `EvalCase.final_session_state` exists in the data model but is completely unused — we can't leverage it for defining expected state in eval sets.

### Proposed API / Implementation

#### 1. Extend `Evaluator.evaluate_invocations`

```python
# evaluator.py
class Evaluator(ABC):
def evaluate_invocations(
self,
actual_invocations: list[Invocation],
expected_invocations: Optional[list[Invocation]] = None,
conversation_scenario: Optional[ConversationScenario] = None,
initial_session_state: Optional[SessionState] = None,
final_session_state: Optional[SessionState] = None,
) -> EvaluationResult:
raise NotImplementedError()
```

#### 2. Wire it through `LocalEvalService._evaluate_metric`

```python
# local_eval_service.py — _evaluate_metric_for_eval_case
evaluation_result = await self._evaluate_metric(
eval_metric=eval_metric,
actual_invocations=inference_result.inferences,
expected_invocations=eval_case.conversation,
conversation_scenario=eval_case.conversation_scenario,
initial_session_state=eval_case.session_input.state if eval_case.session_input else None,
final_session_state=session.state, # from session_service.get_session()
)
```

Both values are already available at evaluation time — `session_input.state` from the `EvalCase`, and `Session.state` from the session service (already fetched for `EvalCaseResult.session_details`).

### Additional Context

- `EvalCase.final_session_state` (in `eval_case.py`) already exists as `Optional[SessionState]` but was never connected to the evaluation pipeline. With the proposed interface change, developers could compare the actual `final_session_state` against this golden/expected value in their custom evaluators.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.